Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1459171 Details for
Bug 1601472
FFU: after the ffu procedure, unable to receive metadata for a freshly booted dpdk instance
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
neutron errors from controller
neutron_errors.log (text/plain), 3.05 MB, created by
Ziv Greenberg
on 2018-07-16 13:14:09 UTC
(
hide
)
Description:
neutron errors from controller
Filename:
MIME Type:
Creator:
Ziv Greenberg
Created:
2018-07-16 13:14:09 UTC
Size:
3.05 MB
patch
obsolete
>2018-07-16 10:38:18.405 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:19.429 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:44.473 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:44.477 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.513 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.515 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.555 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.558 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:47.575 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.576 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.588 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.601 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.603 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.612 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.627 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.627 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.651 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.653 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.672 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.673 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.882 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:53.953 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:55.684 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.695 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.683 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.706 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.694 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.705 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.719 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.729 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.742 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.719 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.731 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.741 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.892 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.963 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.754 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:10.765 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.778 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.790 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.801 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.754 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.811 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.764 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.822 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.776 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.833 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.843 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.855 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.905 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.973 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.789 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.801 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.866 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.812 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.877 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.890 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent [req-ff93b494-eebd-4dce-b6ff-89f05197dde2 - - - - -] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 829ca4bec7064c918d18a70ff203e2d8 >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 742, in _report_state >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent retry=self.retry) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent retry=retry) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id) >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 829ca4bec7064c918d18a70ff203e2d8 >2018-07-16 10:39:34.682 665936 ERROR neutron.agent.dhcp.agent >2018-07-16 10:39:34.693 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: None: error: [Errno 32] Broken pipe >2018-07-16 10:39:35.711 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.724 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.744 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.901 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.915 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.756 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.912 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.767 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.923 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.778 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.824 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.835 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.850 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.791 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.802 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.814 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.935 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:51.946 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.958 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.828 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.926 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.970 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.839 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.851 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.981 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.993 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.863 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:58.004 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.877 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.891 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.902 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.016 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.913 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.027 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.865 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.925 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.037 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.875 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.938 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.889 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.937 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.949 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.961 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.048 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.975 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.060 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.070 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.901 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.912 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.986 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.923 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.996 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.007 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.089 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.102 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.113 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.949 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.018 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.029 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:29.040 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:29.937 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.948 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.960 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.127 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.141 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.151 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.962 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.052 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.063 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.073 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.976 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.164 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.990 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.176 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.001 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.189 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.086 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.973 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.097 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.108 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.203 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:03.214 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.015 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.224 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.028 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.236 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.039 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.247 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.258 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.984 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.120 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.268 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.136 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.146 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.279 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.290 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.301 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.312 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.327 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.338 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.999 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.052 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.063 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.077 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.158 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.168 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.180 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.349 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.362 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.373 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.010 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:43.386 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.397 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.408 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.094 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.105 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.195 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.117 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.206 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.217 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.022 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.419 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.430 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:59.442 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.032 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.229 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:10.131 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.241 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.144 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.254 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.155 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.264 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.455 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.281 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.465 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.293 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.476 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.305 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.319 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.332 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:22.346 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.044 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.358 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.370 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.381 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.491 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.504 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.517 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.172 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.183 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.194 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.392 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.054 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.403 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.414 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.426 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.436 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.531 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.447 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.544 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.555 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.065 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.208 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.458 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.219 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.469 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.230 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.480 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.077 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.571 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:14.582 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.597 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.609 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.622 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.636 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.492 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.647 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.502 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.514 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.087 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.661 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.672 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.688 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.699 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.711 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.246 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.726 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.260 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.272 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.526 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.099 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.540 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.551 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:42.737 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.750 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.763 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.112 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.775 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.786 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.798 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:58.562 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:59.573 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:00.589 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.285 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.297 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.308 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.808 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.123 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.820 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.831 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.603 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:21.614 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:22.625 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.134 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.843 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.854 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.864 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.145 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.320 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.332 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.343 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.877 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.888 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.640 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.900 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.654 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.665 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.155 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.912 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.923 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.934 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:09.166 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.677 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.693 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.704 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.356 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:15.370 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:16.388 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.177 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.946 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.957 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.968 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.717 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.188 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.727 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.741 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.405 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.984 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.419 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.994 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.432 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.004 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.198 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.753 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.209 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.769 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.783 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:15.020 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:16.034 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:17.046 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.446 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.461 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.220 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.472 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:39.231 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:40.799 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:41.810 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:42.821 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.059 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.831 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.069 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.079 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.845 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:46.861 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:47.875 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.892 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.903 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.913 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.242 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.484 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.494 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.506 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:59.926 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.937 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.952 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.255 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.966 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.976 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.989 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.099 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.112 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.128 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.005 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:23.016 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.027 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.266 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.521 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.532 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.543 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.040 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.052 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.063 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.277 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.146 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:46.157 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.168 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.178 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.193 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.204 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.077 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.216 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.089 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.100 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.287 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.228 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.240 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.250 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.263 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.554 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.275 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.565 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.287 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.576 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.299 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.111 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.121 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.132 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.305 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.317 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.328 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.312 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.342 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.357 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.369 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:30.145 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:31.156 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.168 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.589 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.323 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.599 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.380 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.612 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.390 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:42.400 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:52.180 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:53.191 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.201 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.333 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.412 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.423 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.435 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:09.345 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:12.626 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:13.636 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.447 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.647 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.463 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.215 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.474 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.227 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.238 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.357 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.488 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.502 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.513 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.374 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.254 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.266 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.277 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:46.661 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:47.672 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.683 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:54.385 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.524 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.536 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.547 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.402 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.290 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.301 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:12.311 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.560 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.696 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.572 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.705 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.582 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.716 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:24.413 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.427 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:40.325 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.338 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:42.349 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:46.595 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.608 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.621 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.438 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.731 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.744 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.755 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.449 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:12.359 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:13.369 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:14.380 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:14.634 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:15.645 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.657 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.459 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.775 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.786 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:30.798 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.471 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:44.672 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:45.683 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.393 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.693 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:47.407 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.422 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.488 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:02.814 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.826 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:04.836 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:09.504 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:16.706 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:17.721 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.734 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:20.440 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:21.456 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:22.467 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.518 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:36.852 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:37.863 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:38.873 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.530 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:50.751 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:51.762 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:52.774 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:54.480 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:54.540 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:55.496 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.509 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:09.554 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:10.886 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.897 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:12.907 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:24.568 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:24.791 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:25.805 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.819 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:28.526 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:29.540 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:30.550 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.585 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:44.925 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:45.936 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:46.947 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:54.596 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:58.837 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:59.853 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:00.865 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:02.566 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.578 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:04.591 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.609 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.961 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:19.974 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.987 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:24.621 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:32.879 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.893 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:34.903 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.607 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:37.618 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:38.630 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:39.633 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:53.002 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.016 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.647 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.027 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.920 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:07.934 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:08.946 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:09.661 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:10.647 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.663 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:12.674 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.671 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:27.041 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.054 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.066 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.687 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:40.961 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.972 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.983 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:44.695 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:45.708 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:46.720 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.733 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.745 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.758 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.773 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:54.699 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.784 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.798 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.809 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.083 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.097 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.108 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.820 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.833 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.847 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.710 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.860 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.872 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.999 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.887 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.012 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.025 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.724 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.898 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.910 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.923 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.122 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.132 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.145 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.738 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.936 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.947 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.958 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:49.037 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:50.048 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.058 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.069 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.080 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.754 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.091 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.971 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.104 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.984 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.996 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:00.117 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.127 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:07.138 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:09.158 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:09.765 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:10.170 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.154 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:14.012 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.183 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 32 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:19.170 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:20.024 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:20.184 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.035 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:24.775 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.199 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.216 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.233 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.048 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.788 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.060 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.249 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.070 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.260 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.270 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.207 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.218 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.228 665936 ERROR oslo.messaging._drivers.impl_rabbit [req-6a989690-6896-4085-aaec-f304fde9a9dc - - - - -] [25ebf9cf-8ece-4633-9ca7-f3089a2fa80b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.282 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.799 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.293 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.304 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.083 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.094 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.105 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [0d4b68fc-754d-460d-8f3f-ea2dac8ebd8e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:09.812 665936 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:10.315 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.327 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.338 665936 ERROR oslo.messaging._drivers.impl_rabbit [-] [c90d1948-1aeb-41e3-8090-d00b47f1c6b0] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:45.486 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:46.542 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:47.560 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:48.577 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:50.593 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:51.621 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:52.645 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:56.668 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:57.680 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:58.690 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:04.703 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:05.715 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:06.726 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:14.742 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:16.649 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:17.661 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:27.680 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:28.692 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:29.703 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:41.722 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:42.737 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:43.749 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [91b87f1b-961c-4a42-a089-a49e55d47c8c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:57.899 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [3330c6a8-9051-4bf4-94f0-8a6c0e588d83] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:57.916 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-b109d948-f7e1-4837-bdc6-a49a10d450ea - - - - -] [444bbe10-b95f-4f5a-9343-7a45cdcdf06e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:58.925 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] [3330c6a8-9051-4bf4-94f0-8a6c0e588d83] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:58.929 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-b109d948-f7e1-4837-bdc6-a49a10d450ea - - - - -] [444bbe10-b95f-4f5a-9343-7a45cdcdf06e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:59.945 4044 ERROR oslo.messaging._drivers.impl_rabbit [req-b109d948-f7e1-4837-bdc6-a49a10d450ea - - - - -] [444bbe10-b95f-4f5a-9343-7a45cdcdf06e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent [req-681d4400-3e5b-4a02-b606-15052dbfe8cd - - - - -] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 4d0806f9dca645a89b9fb57793f4c7a1 >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 742, in _report_state >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent retry=self.retry) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent retry=retry) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id) >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 4d0806f9dca645a89b9fb57793f4c7a1 >2018-07-16 11:06:59.958 4044 ERROR neutron.agent.dhcp.agent >2018-07-16 11:07:00.978 4044 ERROR neutron.common.rpc [req-b109d948-f7e1-4837-bdc6-a49a10d450ea - - - - -] Timeout in RPC method get_active_networks_info. Waiting for 23 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 3df4b996ea8e48bba9d1bd1eb84e0a41 >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent [req-b109d948-f7e1-4837-bdc6-a49a10d450ea - - - - -] Unable to sync network state.: MessagingTimeout: Timed out waiting for a reply to message ID 3df4b996ea8e48bba9d1bd1eb84e0a41 >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 189, in sync_state >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent active_networks = self.plugin_rpc.get_active_networks_info() >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 574, in get_active_networks_info >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent host=self.host) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 185, in call >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent time.sleep(wait) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent self.force_reraise() >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent six.reraise(self.type_, self.value, self.tb) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent return self._original_context.call(ctxt, method, **kwargs) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent retry=self.retry) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent retry=retry) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id) >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 3df4b996ea8e48bba9d1bd1eb84e0a41 >2018-07-16 11:07:23.778 4044 ERROR neutron.agent.dhcp.agent >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent [req-d56e5cf9-ac8f-46f4-aa91-ecba9b049e0a - - - - -] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID d96b12b1af104e82814ef157c72c1494 >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 742, in _report_state >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent retry=self.retry) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent retry=retry) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id) >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID d96b12b1af104e82814ef157c72c1494 >2018-07-16 11:07:59.967 4044 ERROR neutron.agent.dhcp.agent >2018-07-16 10:38:20.118 666209 WARNING oslo_rootwrap.client [-] Leaving behind already spawned process with pid 668499, root should kill it if it's still there (I can't): error: [Errno 2] ENOENT >2018-07-16 10:38:20.449 666209 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/qg-d39f1bf9-eb/accept_ra: No such file or directory >.: ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/qg-d39f1bf9-eb/accept_ra: No such file or directory >2018-07-16 10:38:20.485 666209 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/qg-d39f1bf9-eb/forwarding: No such file or directory >.: ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/qg-d39f1bf9-eb/forwarding: No such file or directory >2018-07-16 10:38:20.519 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:21.534 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:44.440 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.498 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.825 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.469 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.519 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.854 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:46.495 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.538 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.865 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.509 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.550 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.878 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.894 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.547 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.571 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.558 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.585 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.907 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.570 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.597 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.919 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.934 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.589 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:55.610 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.600 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.621 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.745 666209 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:56.949 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.611 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.632 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.963 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.622 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.973 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.632 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.649 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.662 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.656 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.669 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.984 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.683 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.995 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.678 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.006 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.690 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.703 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.758 666209 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.698 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:14.722 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.724 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.019 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.743 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.745 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.033 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.766 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.767 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.044 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.781 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.792 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.803 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.788 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.813 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.811 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.837 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.768 666209 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.827 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.851 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.056 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:28.071 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.082 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.093 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.104 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.862 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.117 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.873 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.128 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.884 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.840 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:37.850 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.139 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.862 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.149 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.873 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.162 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.779 666209 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.884 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.903 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.904 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.923 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.924 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 844a00163d6e44179cb9efdc7f259a3b >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent Traceback (most recent call last): >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 753, in _report_state >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent True) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent return method(context, 'report_state', **kwargs) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent retry=self.retry) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent timeout=timeout, retry=retry) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent retry=retry) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent 'to message ID %s' % msg_id) >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID 844a00163d6e44179cb9efdc7f259a3b >2018-07-16 10:39:44.201 666209 ERROR neutron.agent.l3.agent >2018-07-16 10:39:44.210 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: None: error: [Errno 32] Broken pipe >2018-07-16 10:39:44.934 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.223 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.174 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.234 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.185 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.246 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.942 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.196 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.953 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.258 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.967 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.275 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.286 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.950 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.298 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.961 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.983 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.210 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.311 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.971 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.996 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.222 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.321 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.008 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.234 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.333 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.346 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.357 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.023 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.035 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.047 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.245 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.984 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.255 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.995 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.265 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.005 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.370 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.381 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.395 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.059 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.071 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.082 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.277 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.288 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.298 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.016 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:25.407 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.027 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.421 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.039 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.432 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.050 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.062 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.074 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.107 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.108 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.118 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.130 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.125 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.136 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.147 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.313 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:39.325 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.444 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.336 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.457 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.350 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.474 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.362 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.166 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.373 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.178 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.384 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.189 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.145 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:49.155 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.395 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.167 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.406 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.180 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.416 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.191 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.211 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.213 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.230 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.231 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.486 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.243 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.497 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.427 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.509 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.438 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.250 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.449 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.261 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.275 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.258 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.269 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.286 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.461 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.283 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.298 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.472 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.308 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.484 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.527 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.538 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.549 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.323 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.337 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.348 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.497 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.296 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.512 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.308 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.523 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.319 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.362 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.373 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.385 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.535 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.561 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.548 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.573 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.559 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.587 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.332 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.344 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.357 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:43.397 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.408 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.419 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.571 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.584 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.595 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.371 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.383 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.603 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.398 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.614 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.625 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:59.430 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:00.441 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:01.452 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.614 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.626 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.637 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.410 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.422 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.432 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.463 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.476 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.489 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.646 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.656 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.667 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.654 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.668 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.679 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.451 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:37.462 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.501 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.473 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.513 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.486 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.525 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.498 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.510 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.521 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:45.680 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:46.690 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.532 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.701 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.543 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.555 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.699 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:50.709 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.721 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.735 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.748 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.566 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.758 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.580 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.769 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.598 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.541 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:00.554 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.780 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.565 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.791 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.576 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.805 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.587 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.598 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.608 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.617 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.620 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.631 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.818 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.828 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.628 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.840 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.641 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.654 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.719 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.730 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.740 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.652 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.671 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.672 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.853 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.689 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.691 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.864 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.703 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.875 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:28.713 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.726 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.739 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.886 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.701 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.898 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.719 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.909 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.731 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.751 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.763 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:42.776 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.759 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.772 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.921 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.783 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.933 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:46.945 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.745 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.756 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.768 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.795 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.806 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.817 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:00.959 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.971 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.982 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.785 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.799 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.810 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.828 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.842 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.854 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:15.808 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:16.820 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.833 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.995 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.006 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:21.017 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.830 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.843 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.855 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.866 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:29.879 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.890 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.032 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.043 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.053 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.866 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.877 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.902 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.854 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.889 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.912 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.864 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.924 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.875 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:01.066 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.078 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.088 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.936 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.903 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.951 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.914 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.962 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.927 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:23.891 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.902 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.100 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.915 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.112 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:27.122 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:34.974 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.985 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.999 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:37.940 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.954 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.966 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.135 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.146 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:53.159 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.930 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.941 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:59.954 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:01.011 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.024 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:03.035 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:05.978 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.989 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.003 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:19.173 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.186 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.198 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:29.048 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:30.059 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:31.070 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:31.970 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:32.982 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:33.992 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.017 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.029 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:38.040 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.212 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.222 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.233 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:59.083 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.095 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.105 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.012 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:07.027 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:08.038 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:08.055 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:09.066 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.078 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.092 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.105 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.117 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.130 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.144 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.156 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.169 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.246 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:22.257 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:23.270 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.281 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.291 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.180 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.301 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.192 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.313 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.204 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.127 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:32.139 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.326 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.152 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.336 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.163 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.348 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.175 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.190 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.215 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.201 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.226 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.236 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.054 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.360 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.065 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.370 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.076 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.214 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.381 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.227 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.238 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.248 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.266 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.267 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.395 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.292 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.293 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.405 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.304 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.415 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:00.315 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.326 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.338 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.429 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.313 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.439 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.324 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.450 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.337 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.353 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.366 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.090 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.377 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.102 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.113 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.463 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.473 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.485 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.348 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.366 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.377 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.391 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.406 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.419 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.496 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.506 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.519 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.392 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.403 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.415 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:42.430 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.443 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:44.455 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.131 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.141 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.152 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.531 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.541 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:52.552 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.427 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.439 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.450 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:00.468 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.480 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.491 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.565 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.577 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:12.589 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.465 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.476 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.504 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.490 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.517 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.165 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.527 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.177 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.189 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:32.601 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.612 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.623 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.542 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.505 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.557 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.516 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.568 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.528 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.206 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.640 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.217 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.651 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.228 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.663 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:06.579 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:07.590 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.605 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.544 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.555 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.566 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.676 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.688 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:24.698 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.241 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:31.252 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:32.264 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:32.617 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:33.629 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:34.640 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.580 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.590 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.603 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.714 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.724 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:52.736 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:00.653 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:01.666 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:02.678 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:04.278 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:05.289 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:06.300 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.619 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:08.630 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.648 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.750 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.768 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.779 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:30.690 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:31.704 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:32.716 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:38.314 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.325 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.662 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.337 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.674 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.685 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.793 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.806 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.817 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:02.730 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.743 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:04.754 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:12.352 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.363 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.699 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.375 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.709 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:15.721 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.830 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.840 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:28.851 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:36.773 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:37.784 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:38.797 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:46.394 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:47.405 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:47.740 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.418 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.753 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:49.764 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.865 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:01.876 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:02.887 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:10.810 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.822 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:12.834 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:20.432 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.448 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.778 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:22.465 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:22.794 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:23.808 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.902 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.913 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:36.925 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:44.848 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:45.860 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:46.871 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:54.477 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:55.491 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:55.825 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.503 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.836 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.847 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.939 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.949 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:10.960 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.883 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:19.895 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.912 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:28.519 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:29.536 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:29.860 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:30.547 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:30.870 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:31.882 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.977 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.988 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:45.003 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:52.927 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:53.940 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.951 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:02.562 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.572 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.896 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:04.583 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:04.906 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.916 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:17.019 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.029 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.043 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.963 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:27.977 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.990 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:36.598 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.609 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.931 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.620 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.942 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.954 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.058 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.069 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.079 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.009 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.024 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.035 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.636 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.648 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.968 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:12.659 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.979 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.990 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.002 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.018 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.030 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.041 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.056 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.068 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.080 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.092 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:26.103 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.115 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.126 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.137 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.092 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.148 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.105 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.159 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.119 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.050 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:36.061 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.170 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.073 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.181 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.083 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.192 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.101 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.113 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.131 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.124 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.142 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.155 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.205 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.675 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.215 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.689 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.136 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.226 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.704 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.150 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.161 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.170 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.188 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.189 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.237 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.206 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.206 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.251 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.217 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.263 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:04.237 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.277 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.217 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.287 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.229 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:10.252 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:11.262 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.298 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 12 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.240 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 14 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:18.723 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.735 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.277 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.299 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.314 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.745 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 32 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:26.311 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.324 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.340 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.255 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.268 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.284 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.324 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.336 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.348 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.355 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.365 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.376 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.295 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.307 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.319 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.364 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.374 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.385 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:57.758 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.769 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.393 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.780 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [e7fcf70b-f663-4b21-a841-eb28bda6797b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.403 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.416 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.332 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.344 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.356 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [1406ef8e-6b34-4714-b435-c59fb9e26779] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.400 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.411 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.422 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [a0450c60-5028-4070-9388-7f43f31dcd2e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:24.436 666209 ERROR oslo.messaging._drivers.impl_rabbit [-] [98a51aa7-cc0f-4146-ba4d-fb80bbdf4922] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 11:04:53.136 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:54.170 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:55.181 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:56.197 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:58.209 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:59.219 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:00.230 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:04.242 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:05.252 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:06.263 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:12.276 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:13.295 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:14.311 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:23.204 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:24.215 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:25.226 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:35.252 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:36.265 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:37.275 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:49.303 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [a9173e08-2492-47d1-a98c-ce6541f31b92] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:50.336 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [89a84309-ee4d-4055-bb13-5730dd9c5952] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:51.347 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [89a84309-ee4d-4055-bb13-5730dd9c5952] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:52.359 5236 ERROR oslo.messaging._drivers.impl_rabbit [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] [89a84309-ee4d-4055-bb13-5730dd9c5952] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:53.377 5236 ERROR neutron.common.rpc [req-2b3a8c24-45b9-438c-81de-5e9f850c29dd - - - - -] Timeout in RPC method get_service_plugin_list. Waiting for 7 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID a6f8e299931b434483c261af1e6b3db6 >2018-07-16 11:09:00.528 5236 ERROR neutron.common.rpc [req-955ee908-018b-4d54-9b8c-3b37c18b1fba - - - - -] Timeout in RPC method get_service_plugin_list. Waiting for 5 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 6ecd0464005c49f1938b9e784b790f21 >2018-07-16 10:38:44.823 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.846 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.859 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.873 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.886 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.898 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.915 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.205 666067 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:55.928 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.947 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.960 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.973 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.983 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.995 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.349 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: None: error: [Errno 32] Broken pipe >2018-07-16 10:39:07.359 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.370 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.381 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.391 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.402 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.416 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.011 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:15.021 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.033 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.048 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.427 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.437 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.060 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.449 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.071 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.083 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.095 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.464 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.109 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.476 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.121 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.489 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.133 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.144 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.154 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.503 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.517 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.528 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.168 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.179 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.192 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.544 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.554 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.565 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.207 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.218 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.229 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.579 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.590 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.602 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.243 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.254 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.265 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.615 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.626 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.636 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.285 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:26.297 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.309 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.320 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.331 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.343 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.354 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.656 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.366 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.673 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.378 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.684 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.389 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.402 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.413 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.425 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.439 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.450 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.699 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.461 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.711 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.721 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.475 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.486 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.496 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.740 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.752 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.768 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.514 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.526 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.537 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.556 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.567 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.580 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:41.785 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:42.796 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:43.813 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.603 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.614 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.625 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.833 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.845 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.857 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.639 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.650 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.661 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.876 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.678 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:36.889 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.688 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.901 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.699 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.709 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.720 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.731 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.742 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.755 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.765 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.776 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.787 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.798 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.812 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.825 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.918 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.836 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.929 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.847 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.940 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.861 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.873 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.883 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.899 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.911 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.923 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.957 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.968 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.980 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.938 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.949 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.962 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.979 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.992 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.004 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.003 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:13.014 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:14.025 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.032 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.043 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.054 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.051 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.064 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.079 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.079 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.092 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.102 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.119 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.130 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.142 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.102 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.113 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:22.123 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.162 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.173 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.184 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.148 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.158 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.170 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.202 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:07.213 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.223 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:28.194 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:29.207 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:30.218 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.239 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.250 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:38.263 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.243 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.254 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:04.270 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:08.280 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:09.290 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.302 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.312 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.326 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.337 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.354 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.366 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.377 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.388 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.402 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.415 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.427 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.303 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.314 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.443 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.325 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.456 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.466 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.481 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.492 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.501 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.518 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.531 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.542 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.352 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.363 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.374 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.557 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.568 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.579 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.596 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.607 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.618 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:44.401 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:45.417 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:46.428 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.636 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.647 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.657 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.457 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.467 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.675 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.478 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.688 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.698 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.715 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.727 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.737 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:52.507 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:53.519 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:54.529 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.757 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.769 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.781 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.561 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.572 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.584 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.799 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.810 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.823 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:00.614 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:01.626 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:02.636 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.842 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:08.853 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.865 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:34.671 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:35.682 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:36.693 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.883 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.894 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.913 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:08.726 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:09.743 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:10.754 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.932 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.945 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:15.955 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:42.789 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:43.803 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:44.818 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:47.975 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.989 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:50.000 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:16.851 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:17.868 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:18.880 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:22.021 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:23.032 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:24.043 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:50.919 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:51.933 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:52.944 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.067 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.078 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:58.089 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:24.964 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:25.982 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.994 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:30.108 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:31.119 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:32.130 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:59.016 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:00.027 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:01.039 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:04.149 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.162 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.175 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.059 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:34.072 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:35.082 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.197 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.209 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:40.220 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.104 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.117 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.127 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.240 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:13.251 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.264 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.275 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.287 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.299 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.310 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.322 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.334 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.345 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.357 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.368 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.380 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.147 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.393 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.158 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.404 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.169 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.415 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.430 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.441 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.453 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.468 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.479 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.493 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.189 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.204 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:22.214 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.509 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.524 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.541 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.561 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.576 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.587 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.233 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.244 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.255 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [92efdf66-6608-4655-8c31-b93ed76b733d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.605 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.617 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.627 666067 ERROR oslo.messaging._drivers.impl_rabbit [-] [701709a7-70e5-4502-9b4f-433acdf3d5b9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:54.367 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:55.407 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:56.419 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:57.431 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:59.442 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:00.453 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:01.464 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:05.479 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:06.490 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:07.500 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:13.522 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:14.533 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:16.265 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:24.278 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:25.289 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:26.300 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:36.317 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:37.328 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:38.339 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:50.353 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:51.363 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [f9d44cbf-004d-4ef5-a637-2d2cf5c311ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:52.432 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [a5d8777a-953c-44be-92e2-ea9b3ff3b104] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:53.442 6055 ERROR oslo.messaging._drivers.impl_rabbit [-] [a5d8777a-953c-44be-92e2-ea9b3ff3b104] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID c5eca7b4ef14404aa51b1752db4dc75b >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent Traceback (most recent call last): >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 267, in _report_state >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent use_call=self.agent_state.get('start_flag')) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent return method(context, 'report_state', **kwargs) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent retry=self.retry) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent timeout=timeout, retry=retry) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent retry=retry) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id) >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID c5eca7b4ef14404aa51b1752db4dc75b >2018-07-16 11:06:54.472 6055 ERROR neutron.agent.metadata.agent >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 48a7fe19fcb84a7b9dbd63d0692659da >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent Traceback (most recent call last): >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 267, in _report_state >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent use_call=self.agent_state.get('start_flag')) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent return method(context, 'report_state', **kwargs) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent retry=self.retry) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent timeout=timeout, retry=retry) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent retry=retry) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id) >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 48a7fe19fcb84a7b9dbd63d0692659da >2018-07-16 11:07:54.477 6055 ERROR neutron.agent.metadata.agent >2018-07-16 10:38:18.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:18.433 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:19.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:19.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:44.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.452 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.453 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.480 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.481 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.505 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.509 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.840 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.841 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.842 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.541 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.549 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.550 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.551 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.552 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.553 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.553 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.806 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.806 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.912 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.913 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.914 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.914 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.916 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.616 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.617 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.617 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.620 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.622 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.622 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.828 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.958 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:46.968 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:46.970 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.974 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.471 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.689 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.690 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.690 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.691 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.693 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:47.693 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.847 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.848 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.978 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.023 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.024 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:48.025 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.026 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.027 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.028 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:48.704 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.052 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.484 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.752 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.753 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:49.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:49.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.758 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:49.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.761 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.865 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.866 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.001 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:50.058 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.074 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:50.077 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.496 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.809 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.810 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.894 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.895 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.102 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.103 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.865 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.866 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.867 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.868 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.868 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.917 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.919 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.093 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.094 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.095 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.907 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.911 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.911 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.914 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.025 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.144 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.145 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.148 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.151 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.932 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.933 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.037 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.186 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.187 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.190 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.920 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.961 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.964 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.965 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.966 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.173 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.207 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.521 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:55.918 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.919 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:55.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.940 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:55.989 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.991 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.199 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.200 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.535 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.954 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.956 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.957 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.022 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.024 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.028 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.029 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.224 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.225 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.546 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.956 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.979 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.980 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.051 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.053 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.055 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.199 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.236 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.557 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.008 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.008 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.009 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.073 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.238 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.239 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.018 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.084 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.220 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.255 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.255 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.568 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.982 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.049 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.073 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.247 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.271 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.273 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.583 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.993 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.060 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.102 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.105 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.105 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.106 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.269 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.270 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.593 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.005 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.100 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.139 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.141 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.142 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.142 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.295 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.297 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.969 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.150 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.304 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.981 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.122 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.995 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.236 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.607 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.092 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.093 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.121 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.157 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.307 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.308 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.621 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.109 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.168 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.326 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.635 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.129 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.167 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.322 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.323 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.343 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.344 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.166 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.183 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.185 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.347 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.348 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.043 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.141 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.177 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.202 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.202 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.371 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.371 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.151 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.382 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.166 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.006 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.019 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.140 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.195 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.155 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.157 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.206 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.268 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.674 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.150 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.173 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.216 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.278 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.360 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.363 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.160 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.184 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.379 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.380 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.056 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.170 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.223 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.416 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.419 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.204 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.243 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.244 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.413 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.437 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.438 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.078 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.217 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.261 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.262 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.425 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.455 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.229 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.436 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.204 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.689 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.192 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.700 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.044 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.203 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.057 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.262 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.199 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:29.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:29.211 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.314 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.440 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:29.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:30.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.223 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.325 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.461 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.461 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.090 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:31.208 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.281 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:31.284 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:31.337 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.493 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:31.496 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.496 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.498 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:32.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.218 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.245 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:32.299 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.447 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.528 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.529 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.531 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.112 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.230 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.349 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.554 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.555 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.123 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.230 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.241 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.268 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.338 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.338 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.361 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.474 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.540 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.550 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.572 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.573 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.373 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.550 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.561 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.135 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.263 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.356 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.357 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.573 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.585 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.599 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.599 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.727 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.146 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.239 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:37.278 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:37.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.620 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.621 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.743 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.159 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.288 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.394 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.639 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.640 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.273 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.313 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:40.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.312 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.403 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.585 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.595 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.092 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.300 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.414 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.596 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.109 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.171 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.324 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.325 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.326 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.616 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.657 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.658 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.120 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.183 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.348 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.349 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.351 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.438 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.439 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.194 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.372 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.373 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.374 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.696 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.698 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.386 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.142 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.498 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.343 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.402 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.402 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.438 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.624 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.627 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.303 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.422 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.449 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.642 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.211 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.366 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.445 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.446 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.480 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.666 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.718 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.719 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.765 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.164 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.222 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.409 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.511 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.513 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.738 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.738 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.775 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.234 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.531 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.756 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.758 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.186 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.432 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.469 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.465 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.484 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.378 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.477 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.503 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.688 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.198 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.291 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.488 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.708 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.708 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.303 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.405 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.551 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.553 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.726 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.728 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.220 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.443 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.570 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.797 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.455 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.521 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:02.591 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.592 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.817 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.316 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:03.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.531 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.542 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.340 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.556 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.525 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.526 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.809 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:07.545 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.547 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.577 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.832 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.233 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.381 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.502 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.587 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.842 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.245 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.416 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.513 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.259 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.774 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.775 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.858 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.278 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.440 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.609 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.610 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.794 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.837 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.838 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.868 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.477 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.601 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.632 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.632 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.856 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.857 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.879 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.408 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.487 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.612 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.650 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.497 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.623 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.316 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.429 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.338 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.587 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.587 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.901 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.918 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.636 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.273 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.641 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.284 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.655 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.295 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.453 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.667 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.468 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.469 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.545 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.556 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.929 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.313 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.489 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.670 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.852 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.854 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.892 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.893 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.942 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.324 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.695 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.696 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.912 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.913 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.952 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.335 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.521 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.930 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.931 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:29.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.684 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.487 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.657 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.659 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.701 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.498 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.679 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.712 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.697 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.698 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.306 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.964 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.980 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.330 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.364 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.995 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.569 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:40.508 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:40.580 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.871 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:40.872 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:41.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.591 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.889 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.347 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:42.533 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.602 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.733 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:42.735 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:42.908 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.951 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:42.952 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:43.358 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.552 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.554 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:43.723 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.750 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.932 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.934 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.968 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.969 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.529 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.564 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.613 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.734 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.768 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.769 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.986 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.987 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.381 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.541 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.562 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.575 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.624 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.750 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.789 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.790 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.958 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.959 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.005 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.007 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.551 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.572 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.585 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.636 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.985 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.986 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.584 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.004 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.030 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.033 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.403 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.596 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.717 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:48.719 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:48.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.845 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.060 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.060 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.061 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.414 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.737 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.863 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.086 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.087 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.088 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.617 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.647 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.342 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.596 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.658 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.775 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.027 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.028 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.352 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.608 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.669 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.048 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.049 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.364 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.425 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.794 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.879 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.880 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.066 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.115 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.116 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.437 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.628 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.820 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.905 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.906 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.137 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.137 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.450 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.639 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.839 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.840 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.154 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.156 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.655 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.762 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.389 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.563 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.681 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.775 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.400 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.575 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.692 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.859 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.860 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.087 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.088 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.411 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.586 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.648 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.703 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.880 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:00.881 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.107 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.108 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.462 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.658 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.900 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.942 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.944 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.125 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.472 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.666 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.961 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.962 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.114 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.194 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.484 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.987 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.988 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.125 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.213 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.214 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.688 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.917 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.918 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.717 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.936 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.379 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.731 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.954 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.955 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.144 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.145 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.390 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.688 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.742 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.403 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.501 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.187 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.238 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.238 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.715 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.029 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.030 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.725 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.053 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.054 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.275 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.276 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.601 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.612 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.822 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.623 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.974 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.975 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.138 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.993 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.995 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.149 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.016 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.160 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.753 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.765 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.210 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.210 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.735 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.434 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.746 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.081 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.250 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.293 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.445 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.548 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.098 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.312 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.313 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.559 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.765 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.126 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.333 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.334 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.777 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.415 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.036 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.061 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.064 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.438 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.635 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.647 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.854 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.659 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.790 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.800 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.172 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.268 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.269 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.184 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.286 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.288 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.573 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.796 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.145 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.146 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.306 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.307 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.354 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.586 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.804 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.167 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.168 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.376 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.376 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.598 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.190 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.191 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.395 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.397 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.825 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.107 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.108 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.125 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.155 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.156 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.458 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.471 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.824 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.451 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:51.810 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.834 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.866 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.334 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.335 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.461 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.824 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.846 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.878 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.472 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.614 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.889 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.211 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.212 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.384 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.385 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.421 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.422 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.492 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.698 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.849 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.230 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.439 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.440 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.860 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.213 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.463 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.465 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.872 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.224 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.515 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.238 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.527 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:00.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:00.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:01.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:01.198 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.216 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.538 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:03.549 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:04.560 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.861 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.859 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.873 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.405 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.406 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.570 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.869 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.884 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.581 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.649 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.880 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.269 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.270 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.445 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.446 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.486 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.594 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.663 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.891 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.506 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.507 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.675 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:13.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.310 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.311 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.524 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:14.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.928 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.936 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.747 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.235 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.236 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.250 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:18.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.261 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.262 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.264 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.758 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.973 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.299 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.304 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.770 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.317 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.616 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.780 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:22.628 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:23.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.340 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.998 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.803 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.022 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.895 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.382 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.464 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.465 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.906 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.922 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.916 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.335 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.339 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.502 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.549 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.550 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.639 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.697 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.945 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.354 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.355 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.570 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.707 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.825 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.955 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.043 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.373 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.373 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.598 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.600 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.665 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.056 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.850 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.404 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.320 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.417 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.337 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.339 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.364 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.075 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:44.087 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:44.873 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:45.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:45.887 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:46.551 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:46.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.439 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.563 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.689 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.450 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.573 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.705 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.460 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.934 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:51.932 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:51.946 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.526 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:52.527 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:52.943 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.956 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.545 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.546 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.721 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:53.956 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.970 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.404 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:54.406 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:54.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.572 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.623 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:54.624 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:54.731 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.967 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.983 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:55.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.424 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.590 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.591 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.641 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.642 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.743 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.982 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.993 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.121 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.660 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.661 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.978 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.992 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.004 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.134 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.468 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.611 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.611 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.680 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.921 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.988 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.002 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.631 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.632 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.765 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.001 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.494 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.495 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.496 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.649 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.650 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.703 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.704 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.025 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.391 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:00.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:00.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.725 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.788 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.036 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.410 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.412 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.549 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.551 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.552 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.748 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.749 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.439 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.439 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.731 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.026 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.680 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.681 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.743 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.025 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.036 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.701 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.036 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.481 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.574 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.575 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.719 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.720 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.771 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.810 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.061 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.499 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.593 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.594 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.790 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.791 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.820 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.071 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.519 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.520 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.611 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.613 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.812 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.082 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.147 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.049 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.158 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.047 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.061 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.169 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.539 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.540 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.741 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.058 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.559 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.559 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.768 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.769 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.831 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.593 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.594 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.595 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.632 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.634 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.830 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.831 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.842 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.094 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.605 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.657 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.657 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.858 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.860 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.105 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.618 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.876 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.877 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.117 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.605 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.641 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.644 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.084 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.673 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.674 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.788 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.095 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.690 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.691 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.812 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.092 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.105 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.841 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.885 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.103 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.705 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.706 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.907 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.128 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.935 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.938 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.140 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.742 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.743 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.953 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.955 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.154 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.194 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.982 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.206 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.994 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.650 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.721 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.721 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.662 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.738 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.674 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.758 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.759 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.123 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.114 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.134 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.883 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.883 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.126 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.148 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.955 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.137 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.778 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.779 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.928 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.929 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.981 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.985 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.818 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.819 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.033 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.187 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.853 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:42.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:42.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:42.876 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.218 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.834 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.020 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.229 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.240 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:46.042 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:46.159 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.150 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.168 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.689 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.950 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.162 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.702 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.968 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.969 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.039 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.847 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.849 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.988 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.991 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.074 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.077 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.077 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.199 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.870 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.872 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.108 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.110 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.110 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.210 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.889 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:52.129 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:52.130 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:52.224 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:52.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:53.712 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.725 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.865 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.884 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.885 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.904 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.905 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.190 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.185 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.200 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.922 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.010 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.059 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.210 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.264 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.933 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.120 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.208 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.276 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.920 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.921 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.051 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.149 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.151 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.235 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.938 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.939 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.177 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.178 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.734 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.956 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.957 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.198 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.200 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.746 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.759 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.934 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.936 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.954 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.955 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:13.981 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:13.982 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.222 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:21.221 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:21.233 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:22.068 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:22.070 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:22.232 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:22.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.087 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.088 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.198 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.243 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.995 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.109 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.110 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.224 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.227 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.271 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.015 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.016 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.255 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.293 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.095 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.276 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.313 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.313 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.743 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.961 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.106 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.324 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.756 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.117 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.768 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.983 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:29.775 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.012 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.791 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.037 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.039 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.803 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.055 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.056 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.264 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.264 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.275 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.133 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.134 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.274 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:42.288 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.151 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.272 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.284 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.063 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.065 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.169 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.171 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.306 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.311 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.312 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.326 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.083 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.085 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.343 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.346 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.346 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.109 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.110 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.372 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.374 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.344 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.079 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.141 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.355 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.368 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.118 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.118 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:53.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.007 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.825 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.018 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.837 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:00.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:01.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.327 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.810 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.297 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.336 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.188 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.308 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.347 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.208 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.320 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.354 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.133 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.135 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.227 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.368 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.398 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.400 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.401 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.378 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.425 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.461 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.171 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.172 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:15.391 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:16.204 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:16.406 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:17.223 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:17.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:18.234 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:19.848 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:22.044 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:23.057 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.362 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:27.337 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:27.372 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.351 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.386 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.220 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.222 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.297 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.298 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.415 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.494 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.495 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.495 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.241 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.429 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.525 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.526 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.527 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.263 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.265 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.550 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.552 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.553 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:34.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.840 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.227 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.856 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:37.245 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:37.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.267 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:43.440 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:44.274 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:44.453 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:45.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:45.464 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:46.303 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:47.885 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.071 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.083 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.400 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:53.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:53.377 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:53.416 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.320 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.320 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.388 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.339 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.340 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.359 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.361 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.578 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.580 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.581 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.311 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.312 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.470 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.607 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.607 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.333 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.333 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.638 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.639 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.639 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.291 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:03.308 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:03.310 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:04.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:04.329 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.869 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.884 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.896 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:13.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:14.332 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:14.497 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:15.344 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:15.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:16.359 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:17.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.939 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:19.950 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.440 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.417 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.388 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.469 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.113 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:23.404 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.405 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.444 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.123 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.362 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.364 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.422 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.668 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.669 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.670 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.134 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.381 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.382 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.511 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.696 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.148 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.400 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.400 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.730 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.730 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.732 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:28.159 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:29.170 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:30.181 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:30.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:30.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:31.368 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:31.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:32.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:32.388 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:34.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:35.204 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:42.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:42.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.920 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.248 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.934 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:46.378 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:46.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:47.390 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:47.544 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.557 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.963 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:50.481 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.568 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.977 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.493 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.578 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.259 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.450 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.452 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.452 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.503 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.589 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.001 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.486 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.487 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.488 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.488 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.291 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.424 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.506 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.508 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.758 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.544 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.794 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.794 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.022 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.460 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.601 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.819 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.034 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.505 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.613 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:59.528 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.408 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.046 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.425 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.057 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.444 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.444 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.069 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:04.305 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:04.638 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.317 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.539 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.333 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.549 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.668 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:07.560 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.091 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.103 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.680 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.691 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.586 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.706 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.952 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.597 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.967 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.346 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.979 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.116 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.358 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.146 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.518 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:23.529 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:23.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.560 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:24.565 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:24.565 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.602 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:25.603 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.603 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:26.481 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:26.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.645 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.646 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.646 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.720 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.848 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:26.850 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:26.851 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:27.497 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.498 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.611 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.669 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.671 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.673 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.673 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.730 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.893 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.893 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.516 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.516 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.629 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.692 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.693 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.694 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.742 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.922 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.923 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.925 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.535 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.640 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.719 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.722 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.722 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.948 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.950 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.951 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.747 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.752 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.752 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.159 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.554 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.554 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.790 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.979 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.982 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.983 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.169 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.465 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:32.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:32.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.572 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.801 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.005 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.007 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.008 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.180 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.484 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.589 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.590 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.030 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.033 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.382 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.766 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.796 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.816 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.817 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.814 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.817 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.837 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.540 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.541 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.608 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.828 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.857 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.857 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.055 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.058 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.058 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.560 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.561 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.839 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.103 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.104 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.578 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.579 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.644 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.653 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.804 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.805 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.826 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.596 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.598 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.675 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.816 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.876 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.616 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.617 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.855 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.856 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.902 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.903 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.636 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.666 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.667 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.867 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.920 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.921 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.164 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.164 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.207 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.688 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.877 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.188 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.190 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.191 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.219 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.709 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.709 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.216 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.999 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.653 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.655 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.831 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.421 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.870 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.024 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.432 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.690 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.690 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.855 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.880 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.940 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.941 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.443 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.895 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.896 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.957 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.958 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.726 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.728 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.908 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.975 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.975 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.240 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.746 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.746 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.918 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.267 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.269 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.270 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.697 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.763 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.764 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.826 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.293 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.295 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.296 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.707 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:59.718 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.233 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.716 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.244 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.732 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.733 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.258 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.762 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.763 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.887 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.915 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.897 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.925 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.993 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.994 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.941 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.943 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.012 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.781 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.782 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.953 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.030 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.321 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.322 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.964 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.347 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.348 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.826 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.373 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.375 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.782 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.478 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.802 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.804 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.848 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.820 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.860 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.756 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.870 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.767 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.913 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.925 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.962 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.271 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.935 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.048 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.049 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.283 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.987 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.989 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.068 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.296 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.846 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.998 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.086 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.088 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.402 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.403 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.863 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.009 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.882 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.883 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.451 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.454 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.454 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:25.045 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.058 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.869 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.870 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.889 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.946 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.898 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.011 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.502 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.780 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.970 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.020 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.104 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.105 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.513 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.796 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.925 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.042 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.043 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.133 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.908 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.909 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.057 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.154 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.155 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.484 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.928 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.068 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.308 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.511 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.945 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.946 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.318 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.540 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.541 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.542 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.329 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.915 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.916 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:44.940 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:44.941 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:45.964 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:45.965 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:52.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:53.064 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.012 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.076 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.177 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.178 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.100 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.196 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.197 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.977 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.979 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.116 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.216 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.568 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.569 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.129 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.592 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.593 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.594 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.016 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.547 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.621 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.838 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.094 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.558 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.851 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:00.106 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:00.568 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.119 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.988 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.357 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.012 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.372 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:12.039 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:13.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:13.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.063 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.138 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.235 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.153 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.050 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.052 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.170 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.274 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.275 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.646 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.648 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.648 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.073 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.097 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.098 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.704 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.705 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.706 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.008 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.066 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.020 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.084 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.085 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.888 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.107 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.108 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.589 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:25.388 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:25.600 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.405 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.611 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:27.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.137 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.092 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.148 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.104 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.159 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.114 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.211 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.314 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.314 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.130 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.220 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.331 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.332 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.737 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.738 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.149 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.150 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.763 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.764 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.764 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.168 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.169 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.790 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.137 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.138 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.155 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.156 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:46.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:46.176 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:47.034 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:47.906 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.045 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.916 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:49.065 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:49.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:51.436 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:52.446 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:52.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:53.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:53.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:54.651 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.145 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:59.156 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:59.229 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.167 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.240 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.351 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.352 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.258 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.261 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.369 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.188 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.190 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.269 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.389 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.391 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.818 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.207 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.208 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.845 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.845 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.846 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:07.183 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.208 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.209 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.234 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.238 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.258 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.259 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:15.078 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:15.939 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:16.090 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:16.949 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:17.102 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:17.962 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:19.471 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.489 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.662 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:24.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:24.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:25.195 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:25.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.205 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.288 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.409 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.410 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.304 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.306 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.429 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.432 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.250 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.316 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.446 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.904 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.904 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.905 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.271 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.272 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.326 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.930 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.931 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.933 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.958 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.959 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:34.295 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:34.296 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:35.318 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:35.319 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:36.344 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:36.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.278 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:42.289 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:43.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:45.117 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:45.974 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:46.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:46.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.294 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.996 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.528 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.539 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:52.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:53.305 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:53.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.315 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.337 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.701 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.366 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.491 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.492 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.716 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.331 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.333 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.376 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.510 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.731 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.987 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.989 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:57.351 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:57.352 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:57.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.015 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.370 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:59.042 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:59.045 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:59.046 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:02.367 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:02.369 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.385 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.386 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:04.405 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:04.406 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:15.339 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.317 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.364 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.023 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:19.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:19.339 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.046 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.559 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.349 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.572 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.359 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.583 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.371 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.533 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.419 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.557 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.558 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.409 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.433 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.575 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.577 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.072 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.073 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.073 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:27.444 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.100 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.447 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.750 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.131 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.761 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:30.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:32.434 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:32.435 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:34.480 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:34.481 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:49.381 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:50.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.361 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.402 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.064 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.374 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.075 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.389 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.088 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.410 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.442 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.597 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.454 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.599 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.606 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.609 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.472 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.473 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.627 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.628 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.487 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.641 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.172 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.176 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.523 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:00.199 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:00.200 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:00.201 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:00.543 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:00.543 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:01.223 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:01.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:01.226 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:02.788 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:04.506 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:04.506 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:04.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.526 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.527 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.546 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.547 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:23.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.436 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.414 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.451 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.101 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.426 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.115 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.437 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:28.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:28.444 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:29.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:29.489 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:29.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.677 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.514 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.538 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.711 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.713 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.563 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.565 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.734 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.250 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.251 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.592 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.592 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.593 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:34.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:34.282 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:34.282 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:34.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:34.619 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:35.324 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:35.327 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:35.328 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:36.826 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:37.839 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:38.587 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:38.588 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:38.851 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.608 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.609 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:40.627 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:40.628 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:57.464 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.480 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.454 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.494 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.142 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:01.154 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:01.478 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:02.166 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:02.489 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.505 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.532 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.746 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.544 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.769 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.771 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.555 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.617 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.800 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.643 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.647 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.648 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.817 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.819 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.357 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.358 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.358 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.683 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.686 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.687 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.395 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.397 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.397 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.715 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:09.425 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:09.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:09.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:10.865 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.878 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:12.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:12.667 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:12.890 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:13.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:13.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:14.711 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:14.712 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:31.513 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.525 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.538 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.181 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.512 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.193 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:36.204 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:36.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:37.545 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:37.574 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:37.819 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.556 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.588 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.858 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.859 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.859 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.600 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.721 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.886 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.886 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.887 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.905 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.907 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.784 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.787 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:42.483 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:42.485 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:42.486 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:42.805 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:42.805 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:43.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:43.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:43.518 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:44.915 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:45.927 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:46.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:46.742 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:46.938 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:47.759 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:47.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.778 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.779 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.555 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.567 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.536 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.578 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.217 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.546 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.229 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.561 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:10.240 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:10.575 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.586 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.612 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.898 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.598 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.624 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.910 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.930 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.931 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.635 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.812 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.923 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.953 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.953 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.833 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.835 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.970 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.973 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.562 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.563 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.564 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.874 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:16.594 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:16.597 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:16.598 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:16.901 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:16.902 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:17.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:17.626 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:17.626 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:19.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.798 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.800 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.983 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.823 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.825 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:22.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:22.843 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:39.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.615 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.600 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.627 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.614 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.268 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.625 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:44.282 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:44.640 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:45.665 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:45.666 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:45.941 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.686 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.688 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.953 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.993 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.994 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.701 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.892 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.966 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.016 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.904 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.924 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.926 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.038 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.666 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.667 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.915 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.944 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.945 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.694 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.695 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.695 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.964 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.965 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:51.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:51.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:51.725 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:53.005 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.018 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.868 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.870 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.031 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.887 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.888 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.906 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.906 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:13.649 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.661 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.649 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.674 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.295 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.660 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:17.306 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:17.675 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:18.322 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.702 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.735 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.715 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.731 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.745 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.978 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.726 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.740 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.059 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.059 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.929 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.002 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:22.078 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.078 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.737 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.942 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.985 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.100 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.778 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.779 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.781 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.782 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.954 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.002 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.003 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.025 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.819 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.821 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.823 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.824 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.021 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.021 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.035 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.845 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.846 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:25.847 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:27.053 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:27.056 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:28.074 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.076 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.838 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.925 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.927 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.092 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.093 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.853 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.947 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.948 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.104 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.864 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.970 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:32.115 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.103 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.126 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:34.118 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:34.140 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:35.130 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:36.877 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.891 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.154 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.902 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:40.178 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.140 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.162 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:46.191 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:46.919 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.201 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.934 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.215 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.730 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.946 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.745 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.338 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:51.179 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.353 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.191 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.792 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.202 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.376 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.787 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.802 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:54.799 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:54.815 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:55.122 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.124 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.812 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:55.826 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.977 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.143 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.143 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.399 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.823 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.839 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.990 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.043 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.044 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.162 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:57.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:57.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.411 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.834 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.849 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.878 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.879 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.880 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.002 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:58.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.068 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.181 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.182 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.850 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.915 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:58.917 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:58.917 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:58.960 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.091 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:59.093 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:59.205 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.205 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.865 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.944 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.947 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.948 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.972 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.027 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.111 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.112 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.224 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.225 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.861 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.980 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.981 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.984 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.039 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.132 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.133 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.422 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.875 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.888 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.010 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.012 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.156 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.157 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.246 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.248 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.441 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.885 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.013 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.014 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.055 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.231 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.266 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.453 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.055 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.065 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.066 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.067 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.068 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.070 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.176 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.177 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.286 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.286 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.119 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.122 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:05.123 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.125 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.126 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.126 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:05.195 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.195 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.253 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.900 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.164 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.166 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.166 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.212 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.213 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.914 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.916 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.189 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.938 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.940 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.212 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.212 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.309 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.311 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.951 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.161 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.329 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.330 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.464 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.211 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.212 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.233 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.233 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.255 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.256 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.257 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.300 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.360 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.361 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.475 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.241 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.242 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.285 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.286 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.290 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.291 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.291 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.292 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.488 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.345 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.346 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.348 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.348 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.349 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.021 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.959 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.032 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.980 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.980 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.046 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.002 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.003 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.383 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.385 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.385 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.017 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.253 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.276 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.420 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.423 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.270 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.272 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.287 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.393 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.394 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.395 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.453 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.456 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.457 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.297 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.298 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.302 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.427 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.428 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.431 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.432 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.309 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.468 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.470 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.471 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.471 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.511 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.522 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.352 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.780 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.791 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.024 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.377 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.482 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.482 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.039 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.040 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.500 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.059 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.059 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.488 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.518 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.519 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.071 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.311 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.506 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.507 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.329 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.331 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.528 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.530 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.533 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.534 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.536 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:28.537 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.066 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.569 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.570 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.571 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.574 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.078 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.362 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.612 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.612 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.614 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.615 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.099 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.556 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.567 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.320 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.579 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.331 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.563 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.564 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.342 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.583 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.584 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.075 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.605 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.097 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.097 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.114 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.115 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.391 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.573 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.591 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.371 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.402 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.586 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.602 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.385 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.387 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.414 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.599 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.621 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.648 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.651 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.652 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.653 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.408 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.682 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.683 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.684 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.419 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.717 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.718 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.718 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.719 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.608 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.634 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.635 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.636 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.127 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.663 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.663 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.141 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.684 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:49.152 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.136 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.157 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.157 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.354 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.174 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.176 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.365 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.615 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.642 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.188 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.376 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.430 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.626 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.654 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.446 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.447 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.637 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.664 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.749 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.752 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.754 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.755 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.466 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.784 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.785 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.787 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.805 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.437 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.827 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.828 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.831 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.831 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.448 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.844 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:58.459 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:00.703 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:01.717 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.703 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:06.188 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.678 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:06.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.689 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.725 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.738 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 14 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:08.175 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.649 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.676 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.700 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:09.486 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:10.501 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:10.502 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:11.392 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:11.524 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.206 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:12.207 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:12.407 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.239 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:14.186 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 20 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.252 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:14.660 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.685 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:15.675 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.699 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.859 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:15.860 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:15.861 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:15.862 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:16.473 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.524 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 16 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:16.898 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.900 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.901 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.491 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.537 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 16 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:17.933 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.936 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.936 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.937 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:18.418 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 20 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.751 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:22.765 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:23.504 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 20 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:25.715 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.728 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.739 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:26.780 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.760 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.761 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.776 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.253 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.872 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.279 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.280 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.885 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.296 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.298 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.695 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.714 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.899 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.315 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.540 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.710 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.726 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.557 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.558 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.724 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.739 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.971 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.975 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.977 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.979 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.203 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.584 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.585 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.006 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.007 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.008 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.009 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.214 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.604 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.042 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.045 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.045 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.046 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.225 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.431 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.443 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.454 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.517 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.809 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.530 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.791 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.822 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.542 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.803 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.786 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.813 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.797 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.309 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.330 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.331 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.350 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.736 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.749 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.362 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.613 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.748 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.759 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:53.629 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:53.632 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:53.759 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:53.769 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.080 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.081 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.082 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.082 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.650 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.650 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.121 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.122 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.123 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.125 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.661 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.161 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.163 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.165 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.249 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.260 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d4ea1f7b-7de4-4d5f-adc0-bdcbeb7a70fd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.467 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.835 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.479 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.846 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.911 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.490 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d28b9d8b-4c7f-4e97-8097-5724d1b8dbb3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.828 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.858 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1c57d71c-ad10-40e2-8457-f4aa344ef89b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.924 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:05.839 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:05.935 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ea50bb7-52e5-48fe-851e-9bf441ef2822] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.850 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [6cb56434-6822-46f4-887f-3786d5d485a3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.554 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.811 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.566 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.824 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:09.578 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4324303-bcd0-4ea4-a3e0-0e9cafbddb26] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:09.836 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d719d81e-dc39-4719-97c7-6d03cb410dda] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.363 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.382 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.383 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.401 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [3db2d86c-18a6-4357-aa89-379be00c04de] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.403 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.772 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.413 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f90ed3c6-152c-4732-8aa0-a8c4fb0fcde9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.783 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:15.672 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.201 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.202 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.203 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.203 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.237 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.239 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.239 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.241 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:19.662 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:20.793 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [19dcf21a-0f7b-4784-a28c-39bab3c48d4d] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:20.803 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [aa359b5b-95db-45cb-a147-7186710f70b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:21.683 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2fc8677-285a-467b-8d3f-8d567bddb857] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.274 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [f89215af-c410-4d38-b6f0-b70a74829acf] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.275 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8d8ec38-988b-41ce-95f3-c4e0fcacd2d9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.277 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [4762aa28-4d1a-4521-a031-adb78db4d031] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.278 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [ec2f9f76-dd95-4070-bdee-e9b3e6fb5de4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 22 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.378 666854 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: 2018-07-16T10:58:25Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) >2018-07-16 10:58:25.448 666854 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json] dies due to the error: 2018-07-16T10:58:25Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) >2018-07-16 10:58:25.897 666854 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: 2018-07-16T10:58:25Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) >2018-07-16 10:58:25.897 666854 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json] dies due to the error: 2018-07-16T10:58:25Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) >2018-07-16 10:58:25.917 666854 ERROR oslo.messaging._drivers.impl_rabbit [-] [7b2c15bd-eb63-447d-a9a4-d8fb223569af] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 11:05:02.070 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:03.081 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:04.091 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:05.103 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:07.114 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:08.125 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:09.135 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:13.148 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:14.158 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:16.063 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:22.075 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:23.089 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:24.099 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:32.114 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:33.128 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:34.142 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:44.154 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:45.165 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:46.176 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [427101ce-5754-4cb4-83ca-c616b8daa6b6] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:58.305 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [8c04dc05-e4a4-42db-9d56-c784db67394f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:59.325 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [8c04dc05-e4a4-42db-9d56-c784db67394f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:00.345 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [8c04dc05-e4a4-42db-9d56-c784db67394f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:01.391 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [97bfdbc1-47e7-4ec2-9552-af68a2e27348] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:02.406 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [97bfdbc1-47e7-4ec2-9552-af68a2e27348] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:03.480 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea88645d-2716-46e3-bad4-ef8e79697186] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:04.492 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea88645d-2716-46e3-bad4-ef8e79697186] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:05.870 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [5ea3f2ea-bbc7-497f-aae9-13085bf9f07a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:05.926 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [47db1801-5ee0-4660-9d69-f80872c3cded] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:06.881 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [5ea3f2ea-bbc7-497f-aae9-13085bf9f07a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:06.937 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [47db1801-5ee0-4660-9d69-f80872c3cded] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:07.913 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [f097c3ef-87b0-430a-a56b-d9ae8520fb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:07.977 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [4c424249-1df7-4ac7-b223-52e566aeb4b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:08.924 6599 ERROR oslo.messaging._drivers.impl_rabbit [-] [f097c3ef-87b0-430a-a56b-d9ae8520fb26] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:08.988 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [4c424249-1df7-4ac7-b223-52e566aeb4b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:10.029 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [02dde29b-0868-46fa-a2d0-c52d2a3ea469] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:11.041 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [02dde29b-0868-46fa-a2d0-c52d2a3ea469] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:12.052 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [02dde29b-0868-46fa-a2d0-c52d2a3ea469] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:13.122 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [5f827ac1-ef9d-401e-886c-4d6770951e5e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:14.133 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-8a28be6e-bd68-40c4-a098-3bfe1782e28a - - - - -] [5f827ac1-ef9d-401e-886c-4d6770951e5e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:15.256 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-813c1d7a-5f35-4b75-bd68-b704c5a2b576 - - - - -] [d226b438-fe55-4729-9fb7-74277a1caa06] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:16.267 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-813c1d7a-5f35-4b75-bd68-b704c5a2b576 - - - - -] [d226b438-fe55-4729-9fb7-74277a1caa06] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:17.278 6599 ERROR oslo.messaging._drivers.impl_rabbit [req-813c1d7a-5f35-4b75-bd68-b704c5a2b576 - - - - -] [d226b438-fe55-4729-9fb7-74277a1caa06] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 7b5e2e64d44845a1be98f46f91ab3797 >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 320, in _report_state >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 7b5e2e64d44845a1be98f46f91ab3797 >2018-07-16 11:07:09.944 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent >2018-07-16 11:07:18.379 6599 ERROR neutron.common.rpc [req-813c1d7a-5f35-4b75-bd68-b704c5a2b576 - - - - -] Timeout in RPC method tunnel_sync. Waiting for 49 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 1a65d687d43c483cad8b9ff86f1b4bf0 >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ac33a1b5026f4192a962d87c82c4a094 >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 320, in _report_state >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 93, in report_state >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ac33a1b5026f4192a962d87c82c4a094 >2018-07-16 11:08:09.952 6599 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent >2018-07-16 07:52:19.123 28 WARNING neutron.plugins.ml2.drivers.type_tunnel [req-04afd753-43e1-4055-864c-df018d47d5f5 - - - - -] Endpoint with ip 10.10.126.102 already exists: DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '10.10.126.102' for key 'PRIMARY'") [SQL: u'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameters: {'host': u'controller-0.localdomain', 'ip_address': u'10.10.126.102', 'udp_port': 4789}] (Background on this error at: http://sqlalche.me/e/gkpj) >2018-07-16 07:52:23.583 29 WARNING neutron.plugins.ml2.drivers.type_tunnel [req-c4add4a5-acf4-4aa6-a382-559606c9dced - - - - -] Endpoint with ip 10.10.126.109 already exists: DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '10.10.126.109' for key 'PRIMARY'") [SQL: u'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameters: {'host': u'controller-1.localdomain', 'ip_address': u'10.10.126.109', 'udp_port': 4789}] (Background on this error at: http://sqlalche.me/e/gkpj) >2018-07-16 10:38:18.364 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:18.378 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:18.403 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:18.441 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:19.390 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:19.408 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:19.429 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:19.462 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:20.420 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:25.282 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:25.285 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:25.846 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:26.016 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:31.917 31 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db **kwargs) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__ >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self.dispatch.engine_connect(self, self.__branch) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db fn(*args, **kw) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db connection.scalar(select([1])) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return meth(self, multiparams, params) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db compiled_sql, distilled_params >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db None, None) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db conn = self._revalidate_connection() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self.__connection = self.engine.raw_connection(_connection=self) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self.pool.unique_connection, _connection) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db util.reraise(*sys.exc_info()) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 649, in get_connection >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db self._get_server_information() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db packet = self._read_packet() >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db packet_header = self._read_bytes(4) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:33.218 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:35.849 29 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines [req-3d9561c9-4e47-4be3-99fd-bf03ec575674 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:36.349 28 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines [req-ba7053ca-923f-4f56-9b66-6abcb4da7576 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.105 30 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:40.306 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:40.309 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:40.861 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:41.028 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:44.448 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.450 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.450 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.461 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:44.476 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.484 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:44.485 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.497 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:44.808 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.822 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.837 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.481 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:45.524 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.534 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.535 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.537 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.538 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.538 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.539 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.855 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.873 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.876 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.493 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.538 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.601 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.609 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.619 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:46.622 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.623 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.624 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.867 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.903 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.905 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.508 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.551 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.627 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.629 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.657 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.660 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.661 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.661 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.923 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.929 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.931 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.519 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.672 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.577 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.654 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.655 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.692 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.694 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:49.696 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.697 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.938 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.949 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.950 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.418 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:50.530 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.590 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.673 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.674 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.718 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:50.722 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.722 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.723 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.952 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.970 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.971 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.545 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.601 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.694 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.695 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.719 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.752 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.753 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:51.753 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.963 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.992 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.994 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.557 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.709 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.731 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.774 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.776 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.723 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.743 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.788 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.855 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:54.736 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.805 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:54.808 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.306 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:55.327 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.328 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.614 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.705 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.766 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.816 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.874 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.975 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.014 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.015 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.039 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.572 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:56.625 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.717 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.747 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.778 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.835 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.836 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.987 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.035 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.036 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.175 29 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:57.587 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.637 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.730 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.760 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.761 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:57.789 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.847 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.886 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:57.999 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.055 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.056 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.600 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.772 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.773 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.859 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.611 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.785 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.796 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.857 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.624 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.869 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.635 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.783 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.807 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.885 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.886 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.647 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.649 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.743 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.795 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.804 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.817 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.896 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.012 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db rec = pool._do_get() >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db self._dec_overflow() >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return self._create_connection() >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return _ConnectionRecord(self) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__ >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db self.__connect(first_connect_check=True) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:04.027 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:39:04.077 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.078 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.663 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.755 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.806 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.816 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.828 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.907 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.027 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.096 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.099 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:05.434 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.678 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.731 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.767 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.830 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.040 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.118 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.119 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.131 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.661 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.142 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.675 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.840 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.868 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.904 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.687 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.851 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.916 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.154 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.317 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.345 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.346 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.818 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:10.862 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.885 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.931 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:10.932 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.049 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.171 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.831 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.942 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.191 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.193 29 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.842 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.897 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:12.956 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.694 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:13.780 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.848 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.853 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.967 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.053 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:14.132 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.709 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.791 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.859 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.066 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.143 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.701 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:15.721 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.802 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.865 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.871 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.978 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.077 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.154 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.203 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.712 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.733 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.877 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.879 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:16.990 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.088 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.214 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.726 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.888 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.891 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.006 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.225 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.743 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.747 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.902 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:18.911 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:39:18.952 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.101 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.764 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.913 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.964 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.112 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.445 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.742 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.754 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.779 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.977 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.123 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.764 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.902 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.926 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.018 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.776 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.913 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.937 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.028 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.880 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.924 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.949 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.040 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.239 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.791 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.135 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.250 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.329 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.368 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.369 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.809 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.820 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:25.891 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:25.900 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.061 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.147 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.167 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.264 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.792 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.825 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.835 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.904 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.161 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.178 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.203 29 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.805 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.847 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.909 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.916 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.962 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.190 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.816 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.858 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.927 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.974 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.936 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.986 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.051 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.869 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.936 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.951 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.994 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:31.064 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.881 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.947 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.965 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.004 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.077 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.838 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.893 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.961 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.017 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.172 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.851 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.029 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.184 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.276 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.830 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.863 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.195 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.288 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.460 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.754 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.841 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.002 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.040 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.301 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.857 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.904 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.972 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.012 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.050 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.915 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.984 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.023 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.061 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.891 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.927 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.995 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.978 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.092 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.203 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.340 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.386 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.386 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.911 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.989 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db rec = pool._do_get() >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db self._dec_overflow() >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return self._create_connection() >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return _ConnectionRecord(self) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__ >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db self.__connect(first_connect_check=True) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:41.022 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:39:41.073 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.103 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.216 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.002 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.073 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.114 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.215 29 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.228 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.878 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:42.920 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.087 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.206 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.891 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.100 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.218 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.869 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.902 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.938 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.012 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.230 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.883 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.950 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.033 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.036 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:46.057 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:39:46.314 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.894 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.962 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.052 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.053 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.326 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.063 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.337 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.111 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.473 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.765 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.123 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.016 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:52.138 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:52.139 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.028 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.149 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.904 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.040 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.162 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.917 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:54.974 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.051 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.064 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.076 30 ERROR oslo_db.api [req-ba7053ca-923f-4f56-9b66-6abcb4da7576 - - - - -] DB exceeded retry limit.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api Traceback (most recent call last): >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return f(*args, **kwargs) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.076 30 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.076 30 ERROR oslo_db.api self.force_reraise() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.076 30 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.076 30 ERROR oslo_db.api context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.076 30 ERROR oslo_db.api host=host) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.076 30 ERROR oslo_db.api cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.076 30 ERROR oslo_db.api **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.076 30 ERROR oslo_db.api for c in query >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return self._execute_and_instances(context) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.076 30 ERROR oslo_db.api close_with_result=True) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.076 30 ERROR oslo_db.api **kw >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.076 30 ERROR oslo_db.api conn = self.session.connection(**kw) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.076 30 ERROR oslo_db.api execution_options=execution_options) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.076 30 ERROR oslo_db.api engine, execution_options) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.076 30 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.076 30 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.076 30 ERROR oslo_db.api conn = bind.contextual_connect() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api e, dialect, self) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.076 30 ERROR oslo_db.api util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.076 30 ERROR oslo_db.api reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return fn() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.076 30 ERROR oslo_db.api fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.076 30 ERROR oslo_db.api rec.checkin() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.076 30 ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.076 30 ERROR oslo_db.api dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.076 30 ERROR oslo_db.api self.__connect() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api connection = pool._invoke_creator(self) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api return Connection(*args, **kwargs) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.076 30 ERROR oslo_db.api self.connect() >2018-07-16 10:39:55.076 30 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.076 30 ERROR oslo_db.api raise exc >2018-07-16 10:39:55.076 30 ERROR oslo_db.api DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.076 30 ERROR oslo_db.api >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server [req-ba7053ca-923f-4f56-9b66-6abcb4da7576 - - - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 437, in report_state >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server context, agent_state) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server host=host) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server for c in query >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return self._execute_and_instances(context) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server close_with_result=True) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server **kw >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server conn = self.session.connection(**kw) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server execution_options=execution_options) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server engine, execution_options) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server conn = bind.contextual_connect() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server e, dialect, self) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return fn() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server rec.checkin() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.__connect() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server connection = pool._invoke_creator(self) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server return Connection(*args, **kwargs) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server self.connect() >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server raise exc >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.083 30 ERROR oslo_messaging.rpc.server >2018-07-16 10:39:55.077 29 ERROR oslo_db.api [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] DB exceeded retry limit.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api Traceback (most recent call last): >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return f(*args, **kwargs) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.077 29 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.077 29 ERROR oslo_db.api self.force_reraise() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.077 29 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.077 29 ERROR oslo_db.api context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.077 29 ERROR oslo_db.api host=host) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.077 29 ERROR oslo_db.api cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.077 29 ERROR oslo_db.api **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.077 29 ERROR oslo_db.api for c in query >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return self._execute_and_instances(context) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.077 29 ERROR oslo_db.api close_with_result=True) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.077 29 ERROR oslo_db.api **kw >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.077 29 ERROR oslo_db.api conn = self.session.connection(**kw) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.077 29 ERROR oslo_db.api execution_options=execution_options) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.077 29 ERROR oslo_db.api engine, execution_options) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.077 29 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.077 29 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.077 29 ERROR oslo_db.api conn = bind.contextual_connect() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api e, dialect, self) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.077 29 ERROR oslo_db.api util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.077 29 ERROR oslo_db.api reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return fn() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.077 29 ERROR oslo_db.api fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.077 29 ERROR oslo_db.api rec.checkin() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.077 29 ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.077 29 ERROR oslo_db.api dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.077 29 ERROR oslo_db.api self.__connect() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api connection = pool._invoke_creator(self) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api return Connection(*args, **kwargs) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.077 29 ERROR oslo_db.api self.connect() >2018-07-16 10:39:55.077 29 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.077 29 ERROR oslo_db.api raise exc >2018-07-16 10:39:55.077 29 ERROR oslo_db.api DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.077 29 ERROR oslo_db.api >2018-07-16 10:39:55.076 28 ERROR oslo_db.api [req-3d9561c9-4e47-4be3-99fd-bf03ec575674 - - - - -] DB exceeded retry limit.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api Traceback (most recent call last): >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return f(*args, **kwargs) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.076 28 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.076 28 ERROR oslo_db.api self.force_reraise() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.076 28 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.076 28 ERROR oslo_db.api context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.076 28 ERROR oslo_db.api host=host) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.076 28 ERROR oslo_db.api cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.076 28 ERROR oslo_db.api **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.076 28 ERROR oslo_db.api for c in query >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return self._execute_and_instances(context) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.076 28 ERROR oslo_db.api close_with_result=True) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.076 28 ERROR oslo_db.api **kw >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.076 28 ERROR oslo_db.api conn = self.session.connection(**kw) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.076 28 ERROR oslo_db.api execution_options=execution_options) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.076 28 ERROR oslo_db.api engine, execution_options) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.076 28 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.076 28 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.076 28 ERROR oslo_db.api conn = bind.contextual_connect() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api e, dialect, self) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.076 28 ERROR oslo_db.api util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.076 28 ERROR oslo_db.api reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return fn() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.076 28 ERROR oslo_db.api fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.076 28 ERROR oslo_db.api rec.checkin() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.076 28 ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.076 28 ERROR oslo_db.api dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.076 28 ERROR oslo_db.api self.__connect() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api connection = pool._invoke_creator(self) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api return Connection(*args, **kwargs) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.076 28 ERROR oslo_db.api self.connect() >2018-07-16 10:39:55.076 28 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.076 28 ERROR oslo_db.api raise exc >2018-07-16 10:39:55.076 28 ERROR oslo_db.api DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.076 28 ERROR oslo_db.api >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 437, in report_state >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server context, agent_state) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server host=host) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server for c in query >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return self._execute_and_instances(context) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server close_with_result=True) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server **kw >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server conn = self.session.connection(**kw) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server execution_options=execution_options) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server engine, execution_options) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server conn = bind.contextual_connect() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server e, dialect, self) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return fn() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server rec.checkin() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.__connect() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server connection = pool._invoke_creator(self) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server return Connection(*args, **kwargs) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server self.connect() >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server raise exc >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.086 29 ERROR oslo_messaging.rpc.server >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server [req-3d9561c9-4e47-4be3-99fd-bf03ec575674 - - - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 437, in report_state >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server context, agent_state) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.force_reraise() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 330, in create_or_update_agent >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server context, agent_state['agent_type'], agent_state['host']) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 269, in _get_agent_by_type_and_host >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server host=host) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server for c in query >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return self._execute_and_instances(context) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server close_with_result=True) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server **kw >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server conn = self.session.connection(**kw) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server execution_options=execution_options) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server engine, execution_options) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server conn = bind.contextual_connect() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server e, dialect, self) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return fn() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return _ConnectionFairy._checkout(self) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server rec.checkin() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server dbapi_connection = rec.get_connection() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.__connect() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server connection = pool._invoke_creator(self) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server return Connection(*args, **kwargs) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server self.connect() >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server raise exc >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:55.087 28 ERROR oslo_messaging.rpc.server >2018-07-16 10:39:55.096 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: None: error: [Errno 32] Broken pipe >2018-07-16 10:39:55.174 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.252 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.351 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.404 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.405 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.922 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.928 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.986 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.076 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.084 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.107 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.241 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:56.264 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.907 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:56.939 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.999 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.063 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.087 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.118 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.185 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.253 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.275 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.919 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.934 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.078 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.088 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:58.130 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.196 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.266 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.929 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.097 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.101 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.209 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.277 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.943 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.113 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.144 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.163 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.352 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.125 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.156 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.177 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.289 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.363 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.955 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.168 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.189 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.300 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.375 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.966 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.110 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.136 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.222 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.312 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.977 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.122 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.148 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.232 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.501 31 ERROR oslo_db.api [req-80bfdd50-4df6-46d8-9c68-585895a637d1 - - - - -] DB exceeded retry limit.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api Traceback (most recent call last): >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return f(*args, **kwargs) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:40:04.501 31 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:04.501 31 ERROR oslo_db.api self.force_reraise() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:04.501 31 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return f(*dup_args, **dup_kwargs) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1412, in get_ports >2018-07-16 10:40:04.501 31 ERROR oslo_db.api items = [self._make_port_dict(c, fields) for c in query] >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return self._execute_and_instances(context) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:04.501 31 ERROR oslo_db.api close_with_result=True) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:04.501 31 ERROR oslo_db.api **kw >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:04.501 31 ERROR oslo_db.api conn = self.session.connection(**kw) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:04.501 31 ERROR oslo_db.api execution_options=execution_options) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:40:04.501 31 ERROR oslo_db.api conn = engine.contextual_connect(**kw) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api e, dialect, self) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:04.501 31 ERROR oslo_db.api util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:04.501 31 ERROR oslo_db.api reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return fn() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return _ConnectionFairy._checkout(self) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:04.501 31 ERROR oslo_db.api fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:04.501 31 ERROR oslo_db.api rec.checkin() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:04.501 31 ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:04.501 31 ERROR oslo_db.api dbapi_connection = rec.get_connection() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:04.501 31 ERROR oslo_db.api self.__connect() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api connection = pool._invoke_creator(self) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api return Connection(*args, **kwargs) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:04.501 31 ERROR oslo_db.api self.connect() >2018-07-16 10:40:04.501 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:04.501 31 ERROR oslo_db.api raise exc >2018-07-16 10:40:04.501 31 ERROR oslo_db.api DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.501 31 ERROR oslo_db.api >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall [req-80bfdd50-4df6-46d8-9c68-585895a637d1 - - - - -] Fixed interval looping call 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._clean_garbage' failed: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall Traceback (most recent call last): >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 145, in _clean_garbage >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall candidates = self._get_dead_floating_port_candidates(context) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 182, in _get_dead_floating_port_candidates >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return {p['id'] for p in self._core_plugin.get_ports(context, filters)} >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return method(*args, **kwargs) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall ectxt.value = e.inner_exc >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return f(*dup_args, **dup_kwargs) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1412, in get_ports >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall items = [self._make_port_dict(c, fields) for c in query] >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return self._execute_and_instances(context) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall close_with_result=True) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall **kw >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall conn = self.session.connection(**kw) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall execution_options=execution_options) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall conn = engine.contextual_connect(**kw) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall e, dialect, self) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return fn() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return _ConnectionFairy._checkout(self) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall rec.checkin() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall dbapi_connection = rec.get_connection() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self.__connect() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall connection = pool._invoke_creator(self) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall return Connection(*args, **kwargs) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall self.connect() >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall raise exc >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.503 31 ERROR oslo.service.loopingcall >2018-07-16 10:40:05.133 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.159 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.244 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.484 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.779 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.188 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.012 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.112 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.200 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.326 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.989 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.027 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.123 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.211 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.337 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.916 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.956 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.001 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.040 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.134 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.172 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.289 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.348 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.967 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.012 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.183 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.301 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.361 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.422 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.423 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:10.516 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:40:10.935 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.979 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.096 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.143 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.201 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.255 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.312 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.157 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.212 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.267 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.945 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.172 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.225 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.279 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.225 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.236 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.236 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:15.360 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.023 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.249 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.372 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.389 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:17.036 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.221 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.382 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.401 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.047 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.233 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.413 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.246 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.423 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.498 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.791 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.054 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.147 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.185 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.291 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.434 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.066 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.158 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.196 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.302 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.445 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.080 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.170 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.207 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.314 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.457 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:23.928 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.261 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:24.997 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:25.275 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.330 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:25.374 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.397 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.444 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.444 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:25.549 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:40:25.948 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.011 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.061 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.110 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.253 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.296 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.343 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.417 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.022 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.073 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.269 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.270 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.354 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.429 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.470 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.956 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.033 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.086 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.291 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.291 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.366 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.481 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.556 31 ERROR oslo_db.api [req-e0db6534-fe42-40a6-9087-dac9a2b9eef7 - - - - -] DB exceeded retry limit.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api Traceback (most recent call last): >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return f(*args, **kwargs) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:40:28.556 31 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.556 31 ERROR oslo_db.api self.force_reraise() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.556 31 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return f(*dup_args, **dup_kwargs) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 238, in get_agents >2018-07-16 10:40:28.556 31 ERROR oslo_db.api agents = agent_obj.Agent.get_objects(context, **filters) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:40:28.556 31 ERROR oslo_db.api cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:40:28.556 31 ERROR oslo_db.api **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:40:28.556 31 ERROR oslo_db.api for c in query >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return self._execute_and_instances(context) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:28.556 31 ERROR oslo_db.api close_with_result=True) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:28.556 31 ERROR oslo_db.api **kw >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:28.556 31 ERROR oslo_db.api conn = self.session.connection(**kw) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:28.556 31 ERROR oslo_db.api execution_options=execution_options) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:28.556 31 ERROR oslo_db.api engine, execution_options) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:40:28.556 31 ERROR oslo_db.api conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:28.556 31 ERROR oslo_db.api conn = bind.contextual_connect() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api e, dialect, self) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:28.556 31 ERROR oslo_db.api util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:28.556 31 ERROR oslo_db.api reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return fn() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return _ConnectionFairy._checkout(self) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:28.556 31 ERROR oslo_db.api fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:28.556 31 ERROR oslo_db.api rec.checkin() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:28.556 31 ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:28.556 31 ERROR oslo_db.api dbapi_connection = rec.get_connection() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:28.556 31 ERROR oslo_db.api self.__connect() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api connection = pool._invoke_creator(self) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api return Connection(*args, **kwargs) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:28.556 31 ERROR oslo_db.api self.connect() >2018-07-16 10:40:28.556 31 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:28.556 31 ERROR oslo_db.api raise exc >2018-07-16 10:40:28.556 31 ERROR oslo_db.api DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.556 31 ERROR oslo_db.api >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall [req-e0db6534-fe42-40a6-9087-dac9a2b9eef7 - - - - -] Fixed interval looping call 'neutron.plugins.ml2.plugin.Ml2Plugin.agent_health_check' failed: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall Traceback (most recent call last): >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall ectxt.value = e.inner_exc >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*dup_args, **dup_kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 249, in agent_health_check >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall filters={'admin_state_up': [True]}) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return method(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall setattr(e, '_RETRY_EXCEEDED', True) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall ectxt.value = e.inner_exc >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall LOG.debug("Retry wrapper got retriable exception: %s", e) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.force_reraise() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return f(*dup_args, **dup_kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 238, in get_agents >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall agents = agent_obj.Agent.get_objects(context, **filters) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall for c in query >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return self._execute_and_instances(context) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall close_with_result=True) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall **kw >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall conn = self.session.connection(**kw) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall execution_options=execution_options) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall engine, execution_options) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall conn = bind.contextual_connect() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall e, dialect, self) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return fn() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return _ConnectionFairy._checkout(self) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall rec.checkin() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall dbapi_connection = rec.get_connection() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.__connect() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall connection = pool._invoke_creator(self) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall return Connection(*args, **kwargs) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall self.connect() >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall raise exc >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.558 31 ERROR oslo.service.loopingcall >2018-07-16 10:40:29.302 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:29.494 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.046 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.379 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.058 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.391 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.071 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.402 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.223 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.326 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.234 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:34.338 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.246 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.350 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.507 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.510 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.806 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.083 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.311 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.413 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.518 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.094 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.098 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:37.195 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:37.322 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.424 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.443 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.529 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.099 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.107 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.110 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.207 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.333 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.436 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.455 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.940 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.114 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.121 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.217 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.313 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.466 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.124 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.139 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.227 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.324 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.385 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.464 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.464 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.966 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.121 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.337 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.154 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.239 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.313 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:42.966 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.165 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.251 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:43.324 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.118 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.178 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.265 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.336 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.447 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.131 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.353 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.457 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:45.499 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:40:45.541 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.143 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.469 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.553 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.262 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.371 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.373 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.565 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.190 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.273 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.279 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.391 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.392 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.202 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.284 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.291 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.412 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.413 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.214 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.303 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.346 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.522 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.819 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.357 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.480 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.146 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.370 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:52.492 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.158 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.361 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.426 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.503 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.951 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.157 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.169 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.376 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.441 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.481 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.170 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.387 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.395 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.454 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.484 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.484 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.493 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.978 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.134 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.181 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.226 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.316 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.505 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.238 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.327 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.578 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.977 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.250 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.339 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.591 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.604 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:01.465 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.476 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:02.490 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:41:03.298 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:03.427 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:03.487 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.309 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.438 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.321 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.453 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.534 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.831 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.197 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.262 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.334 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.351 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.386 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.464 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.518 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.208 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.273 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.362 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.398 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.518 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.528 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.183 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:08.222 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.284 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.345 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.375 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.409 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.476 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.530 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.539 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.962 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.194 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.357 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.401 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:09.487 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:09.546 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.205 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.370 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.407 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.413 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.499 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.502 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.503 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.992 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.147 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.217 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.424 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.507 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.616 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.438 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.520 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.628 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.988 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.228 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.532 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.638 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.239 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.382 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.449 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.518 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.250 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.395 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.461 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.529 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.406 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.472 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.541 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.297 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.389 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.262 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.308 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.401 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.239 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.276 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.320 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.413 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.483 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.547 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.553 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.843 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.252 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.287 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.495 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.564 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.263 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.421 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:22.489 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:41:22.507 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.554 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.575 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.432 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.550 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.564 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:23.975 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.431 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.447 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.563 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:24.576 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.418 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.442 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.521 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.521 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.559 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:25.574 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.004 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.158 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.454 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:26.571 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.298 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.582 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.652 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.000 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.309 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.520 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.664 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.320 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.531 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.675 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.542 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.334 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.426 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.464 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.587 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.348 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.438 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.475 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.599 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.359 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.451 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.486 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:34.611 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.568 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.854 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.282 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.588 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.293 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.334 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.588 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.598 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.308 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.348 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.556 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.601 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.610 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.986 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.362 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.481 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:41:39.568 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.613 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.428 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.540 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.541 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.580 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:41.016 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:41.169 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:43.014 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.473 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.499 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.624 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.485 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.509 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.601 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.640 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.690 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.496 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.519 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.611 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.651 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.702 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.627 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.718 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.372 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.465 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.374 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.383 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:49.479 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.386 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.398 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.490 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.581 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.592 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.865 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.398 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.604 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.615 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.636 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:53.999 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.329 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.627 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.649 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.346 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.443 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.560 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.560 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.638 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.661 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.028 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.184 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.361 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.649 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.027 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.532 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.669 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.477 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:41:59.543 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:59.682 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:00.555 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:00.693 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:03.410 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:04.423 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:04.629 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.434 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.594 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.641 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.731 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.877 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.412 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.505 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.510 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.653 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.745 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.423 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.516 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.522 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:07.645 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:07.756 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.440 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.529 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.535 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.657 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.016 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:09.668 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.454 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.577 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.578 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:10.679 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.040 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.197 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:11.680 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.692 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:12.693 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.038 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.703 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.705 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.386 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.569 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.663 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.713 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.716 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.397 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.580 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.674 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.724 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.410 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:16.470 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:42:16.593 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.684 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.738 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.728 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.446 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.741 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.457 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.606 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.665 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.753 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.888 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.467 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.676 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:22.689 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.030 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.467 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.603 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.604 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.052 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.208 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.454 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.546 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:26.765 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.465 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.557 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:27.781 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:27.782 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.051 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.477 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.568 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.803 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.803 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.814 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.548 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.826 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.559 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.726 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.573 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.605 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.739 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.752 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.837 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.616 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.751 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.764 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.848 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.633 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.777 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:34.860 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.620 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.901 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.440 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:36.468 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:42:36.700 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:36.823 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.451 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.481 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.711 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.833 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.462 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.492 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.701 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.722 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.846 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.873 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.041 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.474 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.504 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.712 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.733 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.886 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.482 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.623 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.624 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.723 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.897 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.065 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.219 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.486 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.744 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.498 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.756 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.067 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.512 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.767 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:46.910 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.527 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.781 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.923 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.495 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:48.539 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.583 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:48.797 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.859 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.935 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.507 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.550 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.595 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.808 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.872 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.518 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.607 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.632 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.883 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.912 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:51.500 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:42:51.530 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.618 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.647 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.789 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.541 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.628 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.659 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:53.763 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:53.799 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.053 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.557 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.641 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.676 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.776 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:54.813 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.493 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.565 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.570 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.645 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.645 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.652 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.788 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.819 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.077 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.229 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.577 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.586 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.800 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.831 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.948 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.516 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.588 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.598 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.841 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.959 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.077 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.527 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.610 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.736 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.811 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.973 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.541 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.582 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.665 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.747 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.822 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.600 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.679 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.759 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.833 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.612 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.697 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.895 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.906 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.844 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:04.919 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.600 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.647 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.852 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.856 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.924 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.612 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.863 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.868 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.623 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.624 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.710 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.873 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.636 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.726 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.986 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.064 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.649 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.747 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.998 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.504 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.663 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:10.663 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.010 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.089 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:11.240 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:12.880 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.092 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.892 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:14.488 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:43:14.691 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:14.833 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:14.903 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.703 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.851 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.715 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.866 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.638 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.661 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.728 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.761 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.882 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.887 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.651 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.673 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.774 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.897 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.932 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.557 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:19.662 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.685 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.740 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.786 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.896 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.908 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.944 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.570 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.659 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.752 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.781 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:20.907 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.935 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.955 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.581 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.763 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.793 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.918 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.594 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.805 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.915 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.023 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.826 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.927 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.035 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.075 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.606 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.626 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.937 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.046 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.515 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.618 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.637 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.683 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.686 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.775 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.839 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.935 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.102 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.252 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.630 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.649 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.787 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.851 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.946 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.799 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.864 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.959 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:28.103 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.699 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.728 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:43:29.804 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.642 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.712 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.815 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.654 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.677 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.724 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.827 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.876 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.920 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.665 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.689 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.888 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.931 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.700 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.811 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.899 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.944 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.974 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.824 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.955 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.984 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.671 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.837 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.946 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.966 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.995 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.967 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:36.977 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:37.980 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.679 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.990 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.060 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.085 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.690 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.913 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.072 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.525 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.712 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.713 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.713 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.924 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.088 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.113 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.264 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.936 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.115 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.736 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.841 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.850 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.006 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.749 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.854 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.864 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.018 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.760 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.866 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:45.875 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:46.030 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.717 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.958 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.726 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.730 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.969 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.989 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.738 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.742 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.950 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.980 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.001 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.684 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.751 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:50.769 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:43:50.966 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.967 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.013 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.979 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.095 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.665 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.538 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.678 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.730 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.731 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:55.889 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.045 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.126 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.274 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.689 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.903 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.003 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.056 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.100 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.915 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:58.017 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:58.067 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:58.111 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:58.127 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:59.029 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:59.123 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:59.773 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:59.881 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:00.769 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:00.789 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:00.893 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.785 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.804 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.907 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.991 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.796 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.002 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:04.014 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.026 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.697 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.760 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.979 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.994 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.036 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:06.714 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:44:06.777 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.005 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.048 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.788 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.018 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.106 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.932 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.083 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.549 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.749 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.750 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.943 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.095 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.137 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.286 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:11.957 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.107 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:13.138 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:14.808 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:15.819 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:16.028 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:16.830 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.041 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.135 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.816 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.920 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.052 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.147 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.829 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.932 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:19.043 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:19.159 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:19.840 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:19.942 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.057 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.714 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.992 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:21.070 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:23.061 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.075 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:24.117 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.086 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.560 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.767 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.768 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.806 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.970 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.032 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.119 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.149 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.299 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.703 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.816 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.982 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.044 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.129 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.719 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.765 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:44:27.827 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.994 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.057 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.141 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.148 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:28.730 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.845 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.859 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.071 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.873 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:33.084 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:34.094 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:35.730 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:36.004 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:37.855 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:37.960 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:38.868 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:38.972 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.129 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.171 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.880 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.988 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.183 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.571 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.786 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.787 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.161 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.194 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:41.311 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.083 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.104 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.160 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:43.714 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:44:44.005 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.095 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.117 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:44.157 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.017 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.106 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.128 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:45.168 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.030 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.184 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.847 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.072 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.859 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.886 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.085 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.871 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.898 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.096 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.112 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.742 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.910 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.016 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.124 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.135 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.140 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.583 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.805 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.806 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:56.174 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:56.324 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:58.172 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:59.895 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:00.002 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:00.749 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:00.907 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:01.013 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:01.762 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:01.919 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.025 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.773 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.207 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.046 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.197 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.218 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:04.760 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:45:05.058 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.145 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.209 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.229 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.754 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.028 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.071 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.157 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:06.220 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.168 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:08.924 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:09.120 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:09.153 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:09.935 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.132 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.154 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.594 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.823 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.823 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.947 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.142 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.167 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.185 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.337 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.895 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.111 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.178 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.908 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.122 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.184 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.923 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.133 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:20.508 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:45:20.768 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.039 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:23.934 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.047 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.163 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:24.945 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.058 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.606 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.850 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.851 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.956 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.069 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.083 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.201 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.233 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.347 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:27.095 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:27.244 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.108 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.195 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:28.261 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.188 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.243 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.200 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.255 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.961 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.212 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.267 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:31.971 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.195 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.982 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.206 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:34.218 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:34.793 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.783 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.811 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.050 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.831 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:37.156 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:37.944 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.150 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.171 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.955 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.161 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.174 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.182 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.969 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.176 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.617 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.867 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:40.868 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:41.212 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:41.358 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:42.080 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:45:43.206 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.971 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.085 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.121 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.275 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.797 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.985 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.063 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.096 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.134 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.289 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:51.999 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.110 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.150 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:52.300 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.185 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.995 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.227 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.627 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.886 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.886 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.005 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.224 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.244 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.245 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.370 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.016 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.264 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.264 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.281 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:45:58.216 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.276 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:58.292 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:59.303 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:05.814 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:05.991 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.078 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.193 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:07.003 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:07.195 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:07.206 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.013 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.208 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.221 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.848 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.196 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.219 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.866 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.644 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.880 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.908 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:10.909 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:11.236 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:11.381 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:13.229 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:16.167 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:16.319 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:17.178 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:17.330 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.021 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.128 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.189 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.341 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.737 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:46:19.032 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:19.140 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.044 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.151 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.832 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.031 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.092 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.044 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.288 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.060 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.285 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.298 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.207 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.297 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.309 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.308 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.658 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.934 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:25.935 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.248 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:26.393 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:27.319 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:28.241 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:28.332 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:29.344 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:34.488 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:46:35.846 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.036 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.112 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.241 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.048 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.253 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:38.059 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:38.265 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:39.219 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:39.233 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:40.244 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:40.670 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:40.959 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:40.960 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:41.256 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:41.260 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:41.405 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:42.267 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:42.895 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.251 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.906 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.203 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.279 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.354 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:44.919 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.214 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.290 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:45.366 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:46.225 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:46.301 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:46.380 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.062 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.166 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.077 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.078 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.178 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.089 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.090 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.191 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.314 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.322 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.864 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.108 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.124 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.327 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.334 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.340 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.347 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.322 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.235 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:54.333 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.345 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.680 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:55.732 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:46:55.977 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.978 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.272 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.416 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.264 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.352 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:59.371 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:59.372 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.391 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.392 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.405 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.416 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:04.432 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.442 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.884 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.140 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:06.463 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:08.086 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:08.281 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:08.414 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.098 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.246 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.294 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.427 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.110 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.307 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.444 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.481 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.691 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:10.763 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:47:10.994 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.995 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.121 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.283 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.317 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.427 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.496 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:12.511 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.133 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.276 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.328 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.146 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.241 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.340 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.394 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.158 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.254 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.350 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.411 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.269 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.430 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:16.941 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.951 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.522 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.964 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.122 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.171 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.365 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.534 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.106 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:20.136 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.185 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.207 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:20.361 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.377 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.458 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.545 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.899 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.118 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.149 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.152 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.198 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.218 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.372 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.389 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.469 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.132 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.229 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.384 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.483 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:23.146 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:23.242 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:24.260 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.160 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.255 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:25.360 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:25.702 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.017 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.018 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.171 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.266 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.297 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.371 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.443 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.182 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.211 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.277 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.382 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.400 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.222 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.287 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.395 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.413 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.558 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.235 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.425 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.569 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.406 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:30.582 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.194 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.289 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.417 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.205 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.301 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.428 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:32.722 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:47:33.216 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.314 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:34.496 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.508 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.912 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.164 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.441 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:36.521 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.247 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.439 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.454 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.261 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.451 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.467 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.230 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.273 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.275 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.326 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:39.463 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.241 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.337 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.599 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.719 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.037 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.037 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.254 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.309 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.349 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.455 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.611 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:42.623 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:43.298 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:44.478 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.490 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.284 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:46.450 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:46.500 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.295 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.462 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.754 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:47:48.307 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.474 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.268 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.289 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.318 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.364 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.478 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.488 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.281 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.300 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.378 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.489 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.533 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.926 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.979 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.164 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:51.176 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.293 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.313 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.330 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.390 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.499 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.499 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.543 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.994 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.177 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.342 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.405 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:52.510 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:52.555 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.009 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.189 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.354 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.416 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.526 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.202 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.287 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.427 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.516 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.635 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.440 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.529 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.647 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.729 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.060 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.060 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.217 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.321 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.465 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.540 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.659 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.227 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.367 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.453 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.538 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.238 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.309 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.382 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.464 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.548 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:59.394 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:59.478 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:59.562 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.306 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.402 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.255 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.324 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.417 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.265 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.331 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.337 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.428 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.489 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.511 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.280 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.344 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.502 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.521 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.355 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.408 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.513 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.534 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.574 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.937 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.187 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.420 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.557 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:06.585 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.432 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.570 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:07.596 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.570 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:08.585 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.299 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.581 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.714 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:48:10.292 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.596 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.673 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.741 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.081 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.082 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.303 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.334 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.477 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.524 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.685 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.320 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.538 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.697 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.322 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.550 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.349 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.443 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.446 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.608 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.360 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.455 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.457 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:16.623 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.374 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.466 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.468 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.635 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.373 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.546 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.337 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.389 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.557 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.597 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.949 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.199 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.349 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.402 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.564 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.575 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.612 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.362 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.575 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.623 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.587 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.312 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:24.744 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:48:25.031 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:25.752 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.044 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.099 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.101 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.349 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.491 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.058 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.483 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.650 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.333 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.495 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.609 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.660 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.710 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.511 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.624 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.677 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.725 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:30.637 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:30.739 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:31.389 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:31.478 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.373 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.401 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.490 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.384 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.412 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.500 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.599 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.401 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.614 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.625 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.961 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.211 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:36.641 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.426 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.592 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.654 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.438 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.604 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:38.669 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.324 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.449 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.615 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.764 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.122 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.123 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.360 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.501 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.523 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.689 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:42.534 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:42.703 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.344 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.545 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.717 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:46.414 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:46.702 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:48:47.427 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:47.637 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.438 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.650 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.754 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.425 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.515 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.661 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.765 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.436 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.528 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.649 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.778 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.973 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.225 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.448 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.540 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:51.661 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:52.673 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.334 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:54.682 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.697 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.782 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.141 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.141 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.373 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.512 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:56.708 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.470 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.557 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.631 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.729 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.358 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.481 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.567 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.641 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.741 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.071 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.492 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.579 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.652 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:59.752 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:00.083 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.095 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:01.735 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:49:02.451 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.462 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.675 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.476 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.689 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:05.699 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:05.986 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:06.236 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:09.352 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:09.469 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:09.554 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.482 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.568 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.792 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.794 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.160 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.160 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.390 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.493 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.524 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.582 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.807 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:12.820 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:13.370 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.686 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.720 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.591 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.702 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.732 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:15.764 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.602 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.713 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.743 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.774 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.614 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.784 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.514 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.668 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.492 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.528 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.679 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.998 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.247 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.505 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.541 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.691 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.712 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.516 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.725 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:23.692 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:49:23.735 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:24.365 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:25.806 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.178 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.179 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.406 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.535 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:28.381 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:31.506 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:31.598 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:32.519 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:32.610 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.109 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.531 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.628 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.120 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.838 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.132 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.626 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.796 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.848 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.016 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.263 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.638 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.755 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.807 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.859 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.649 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.766 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:37.818 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:49:38.777 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:39.378 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.528 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.726 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:40.817 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.196 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.197 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.417 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.541 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.546 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.737 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.748 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.555 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.749 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.760 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.392 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.572 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.704 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.771 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.584 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.716 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.596 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.729 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:51.029 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:51.279 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:54.390 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:55.544 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:55.644 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:55.828 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.215 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.216 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.435 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.556 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.558 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:56.654 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.568 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.662 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.666 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.831 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.407 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.672 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.840 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:59.684 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:59.852 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:00.484 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:50:00.789 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.871 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.804 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.888 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.569 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.817 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.899 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.580 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.784 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.591 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.795 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:05.809 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:06.042 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:06.298 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:07.148 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.159 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:08.766 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.174 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.402 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.624 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.745 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.776 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.635 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.759 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.788 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:10.839 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.234 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.234 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.447 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.569 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.647 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.772 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:13.417 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:15.713 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:50:21.053 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.311 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.583 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.680 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.696 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.864 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.594 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.692 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.707 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:22.876 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.605 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.705 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.718 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:23.887 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:24.413 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:25.850 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.256 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.257 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.459 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.580 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.606 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:26.832 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.618 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.822 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.843 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.428 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.629 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.833 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.854 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.913 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.845 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.924 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.936 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:36.068 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:36.329 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.471 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:50:37.677 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.785 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.689 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.798 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.809 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.426 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.706 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.810 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.821 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:40.836 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:40.863 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.192 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.283 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.284 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.472 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:41.593 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:42.204 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:43.216 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:43.445 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.731 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.900 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.743 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.912 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.619 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.720 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.754 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.924 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.630 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.731 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.080 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.341 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.642 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.744 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:52.644 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:52.700 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:50:53.655 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:53.858 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.440 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.666 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.883 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.884 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.876 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.902 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.903 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.303 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.304 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.483 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.612 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:56.916 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.456 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.949 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:59.962 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:00.977 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:06.093 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:06.353 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.724 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.825 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:08.736 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:08.838 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.452 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.750 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.852 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:10.851 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:10.887 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:11.322 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:11.324 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:11.498 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:11.623 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:11.862 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:12.875 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:13.466 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:14.460 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:51:15.232 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:15.770 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:15.939 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.246 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.781 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.954 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.261 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.797 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.966 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:19.657 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:19.762 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.670 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.683 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.773 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.104 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.369 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.683 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.701 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.791 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.916 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.714 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.927 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.939 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.465 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:24.935 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.897 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:25.946 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.340 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.340 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.510 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.634 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:26.958 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:28.481 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:51:30.990 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:32.001 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.013 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:36.117 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:36.381 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.477 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.766 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:39.874 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.779 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.886 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.908 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.359 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.359 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.521 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.645 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.790 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.897 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:43.491 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:44.890 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:45.810 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:45.901 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:45.979 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.821 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.913 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.990 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:47.835 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.001 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:49.278 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:50.293 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:50.730 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.133 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.305 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.393 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.698 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.743 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.811 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.961 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.710 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.754 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.822 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.976 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.722 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.834 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.988 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:54.043 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:51:54.488 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.919 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.378 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.378 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.535 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.656 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.984 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.994 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:58.505 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:59.005 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.028 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.039 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:06.072 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:52:06.146 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.404 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:07.051 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:09.507 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:10.937 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:11.396 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:11.397 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:11.549 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:11.672 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.517 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.807 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:13.912 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.818 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.927 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:15.829 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:15.939 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:17.851 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.015 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.865 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.926 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:19.027 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:19.878 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:19.938 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:20.037 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:20.951 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:21.160 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:21.421 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:22.770 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:23.320 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:23.783 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.010 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.336 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.524 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.799 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.022 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.351 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.740 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.852 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.948 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.033 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.415 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.415 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.561 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.683 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.751 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:26.863 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.764 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.875 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:28.528 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:30.469 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:52:31.030 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:32.041 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.057 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:36.172 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:36.432 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.066 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.536 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:40.077 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:40.966 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:41.089 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:41.436 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:41.437 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:41.575 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:41.699 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:43.540 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:45.500 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:52:47.846 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:47.956 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.861 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.967 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:49.872 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:49.978 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:51.189 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:51.448 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:51.899 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:52.054 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:52.911 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:52.964 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:53.064 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:53.927 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:53.978 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:54.080 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:54.552 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:54.989 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:55.979 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.462 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.463 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.589 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.711 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:56.816 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:57.367 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:57.827 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.052 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.384 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.552 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.842 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.063 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.397 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.780 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.893 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.074 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.792 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:00.907 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:01.804 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:01.919 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.075 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:05.338 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:53:06.089 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.202 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.460 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:07.100 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:09.567 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:10.991 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.483 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.483 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.602 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:11.723 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:13.105 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:13.563 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:14.122 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:15.138 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:20.469 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:53:21.219 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.475 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.893 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.995 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:22.905 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:23.006 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:23.917 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:24.018 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:24.581 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:25.946 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.003 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.097 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.502 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.503 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.615 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.736 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.959 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:27.002 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:27.108 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:27.972 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:28.015 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:28.121 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:28.578 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:29.028 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:30.857 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:31.411 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:31.868 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.098 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.423 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.879 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.111 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.435 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.824 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.939 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.123 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.836 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:34.950 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.847 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.966 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:36.232 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:36.486 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.122 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.593 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.134 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.017 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.146 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.521 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.522 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.627 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:41.748 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:43.592 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:44.710 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:53:47.154 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.170 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:49.186 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:51.249 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:51.502 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:54.604 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:55.943 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.032 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.042 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.548 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.549 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.638 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.761 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:56.956 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.052 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:53:57.968 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:58.063 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:58.607 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:59.989 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:00.143 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:01.001 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:01.042 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:01.155 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:02.011 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:02.052 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:02.171 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.067 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:04.896 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.451 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.907 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.153 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.262 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.462 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.514 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.919 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.165 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.476 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.862 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.984 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.176 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.874 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.996 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.615 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.887 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:10.010 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.044 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.568 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.568 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.649 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.774 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.166 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.620 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.176 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:15.192 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.207 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.280 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.528 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:21.712 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:54:22.222 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:23.239 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:24.627 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.059 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.596 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.597 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.665 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:26.791 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:28.636 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:29.986 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:30.077 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:31.000 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:31.088 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:32.011 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:32.100 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:34.031 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:34.190 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:34.457 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:54:35.042 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:35.080 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:35.203 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.054 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.090 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.216 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.293 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:36.541 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:37.103 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:38.937 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:39.492 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:39.638 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:39.948 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.211 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.503 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.960 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.071 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.223 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.514 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.617 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.617 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.677 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.801 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.902 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.036 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.238 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.915 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.047 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.647 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.926 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:44.058 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.223 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.240 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.252 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:51.307 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:51.553 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:54.655 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.255 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.090 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.269 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.639 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.639 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.692 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:56.814 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:57.280 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:58.659 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.699 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:55:04.031 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:04.115 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.043 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.127 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.056 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.137 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.319 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.566 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:08.074 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:08.234 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:09.089 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:09.122 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:09.246 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:09.667 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:10.100 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:10.137 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:10.261 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.101 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.150 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.659 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.660 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.703 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:11.825 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:12.974 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:13.531 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:13.672 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:13.918 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:55:13.987 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.255 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.543 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.998 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.266 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.554 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.944 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.075 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.281 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.955 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:17.090 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:17.968 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.105 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.275 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.330 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.577 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.287 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:23.299 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:24.678 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.113 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.680 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.681 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.716 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:26.835 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:28.686 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.299 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.310 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:31.322 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:34.963 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:55:36.342 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:36.589 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.082 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:38.154 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.094 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.166 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.696 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:40.110 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:40.178 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.122 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.124 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.700 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.701 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.728 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.846 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.115 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.134 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.285 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.128 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.144 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.165 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:43.295 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.701 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:44.140 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:44.176 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:44.307 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:45.155 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:45.187 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:46.166 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:46.200 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.015 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.181 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.570 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.029 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.212 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.305 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.583 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.039 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.223 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.320 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.595 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.984 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.125 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.235 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.330 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:50.470 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:55:51.001 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.135 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.197 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.354 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.600 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.013 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.151 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:52.212 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.227 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:54.247 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:54.709 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.264 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.316 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.136 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.278 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.328 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.719 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.720 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.739 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.856 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.338 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:58.712 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.241 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.253 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.264 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:02.291 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.301 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.337 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:04.314 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:04.348 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.359 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.366 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.371 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.613 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:08.386 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.277 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.397 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.720 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.289 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:10.414 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.153 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.300 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.740 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.740 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.751 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:11.867 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:12.039 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:56:12.197 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:12.326 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.209 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.338 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.729 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.220 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.350 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.428 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.232 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.448 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.158 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.339 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.464 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.169 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.243 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.352 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.180 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.258 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.364 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.268 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.054 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.319 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.380 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.609 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.623 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.068 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.330 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.351 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.478 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.625 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.082 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.280 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.342 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.365 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.491 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.637 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.034 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:24.168 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:24.290 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.364 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.379 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.504 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.731 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.045 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.180 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.304 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.384 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.056 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.164 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.191 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.395 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.758 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.759 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.763 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.883 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.067 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.204 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:56:28.744 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.078 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.217 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.361 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:30.089 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.231 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.372 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.100 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.242 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.316 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.383 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.327 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.394 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.516 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.339 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.527 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.407 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.539 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.113 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.253 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.356 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.419 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.125 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.264 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.367 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.392 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.431 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.636 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.136 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.276 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.378 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.407 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.427 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.744 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.439 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:40.443 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.175 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.351 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.464 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.776 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.780 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.781 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:41.895 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.366 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.476 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.148 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.288 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.388 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.755 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.162 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.304 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.553 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.173 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.315 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.564 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:46.577 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.492 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.689 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:56:49.507 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.195 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:50.395 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:50.521 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.208 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.395 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.404 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.405 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.649 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.220 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.407 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:52.417 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.187 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.233 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.329 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.401 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.418 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.429 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.199 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.340 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.413 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.453 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.756 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.102 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:55.211 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.244 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.353 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.425 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.441 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.466 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.650 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.113 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.186 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.255 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.402 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:56.452 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.478 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.662 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.788 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.800 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.801 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:56.907 27 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.125 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.267 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.414 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.463 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.675 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:58.136 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:58.425 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:58.532 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:58.766 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:59.436 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:59.543 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:00.149 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:01.161 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:01.280 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:01.474 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:02.293 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:02.485 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:03.589 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:04.470 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:57:04.600 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.232 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.372 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.555 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 10 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:05.613 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.414 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.447 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:06.662 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.177 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:07.440 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.457 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.305 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 6 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:08.450 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.468 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.496 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 6 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:09.767 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:11.197 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:11.198 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:11.245 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:11.389 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:12.213 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.261 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.405 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.495 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.509 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:13.784 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:14.315 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:14.436 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.461 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 14 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.507 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.326 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.447 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.518 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.574 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.463 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.588 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:16.805 27 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:16.820 30 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:16.832 30 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:16.918 27 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:17.481 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:18.226 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 6 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:18.499 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.511 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.520 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 18 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.343 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.429 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.531 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.675 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.821 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.846 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.847 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:21.931 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:22.599 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 12 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:24.244 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:24.279 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:24.421 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:24.628 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:24.785 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.255 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.290 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.437 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.523 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.642 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:25.680 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:57:25.843 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.875 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.876 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.955 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.211 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.265 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.302 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.450 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.533 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.657 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.854 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.893 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.894 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.969 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.549 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.873 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.918 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.920 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.984 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.479 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.798 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.359 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.490 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.544 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.688 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.889 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.946 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.947 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.001 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.375 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.505 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.555 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.703 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.900 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.975 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.975 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.017 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.392 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.570 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.719 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.914 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.992 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.993 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.031 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.279 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.489 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.614 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.293 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.500 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.566 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.625 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.931 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.011 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.012 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.045 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.307 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.440 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.519 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.577 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.635 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.687 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:36.943 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.028 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.029 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.059 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.539 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.595 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.954 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.050 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.051 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.071 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:38.550 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.561 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:39.798 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.316 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.463 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:40.710 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:57:41.221 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.328 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.404 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.475 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.586 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.341 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.415 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.487 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.597 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.670 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.426 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.609 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.680 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.813 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.967 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.070 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.071 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.083 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.700 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:44.980 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.093 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.094 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.097 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:45.991 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.108 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.112 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.114 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.319 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.522 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.335 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.540 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.609 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.348 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.554 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.619 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.630 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.650 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.452 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.663 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.698 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.674 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.002 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.120 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.132 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.133 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:54.812 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.014 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.130 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.154 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.155 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.439 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.627 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.025 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.141 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.176 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.177 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.233 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.450 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.540 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:56.638 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:57.461 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:57.551 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:57.652 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.353 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.502 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.562 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.824 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.365 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.514 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:59.576 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.360 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.378 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.528 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.588 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.372 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.600 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [5986e499-bc05-4cf3-bc97-314f96963c7f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.648 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db [req-0c2fea6b-998b-4030-9ad6-d5bcdc6a6b29 - - - - -] Exception encountered during router rescheduling.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 132, in reschedule_resources_from_down_agents >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db down_bindings = get_down_bindings(context, agent_dead_limit) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 76, in get_down_router_bindings >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/l3agent.py", line 65, in get_down_router_bindings >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db query.all()] >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2726, in all >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return list(self) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db conn = engine.contextual_connect(**kw) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:01.753 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:58:02.383 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.659 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.715 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.671 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.737 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.737 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.757 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [719c67d9-409e-4b65-9f9b-848db575e603] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.758 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:05.770 29 ERROR oslo.messaging._drivers.impl_rabbit [req-ca3b32b5-cfc1-416e-8953-71f6a953244c - - - - -] [3edcec68-1c4b-4838-bbc8-c075b66d342a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.040 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.157 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.199 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.200 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.464 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.567 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.712 28 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.054 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.170 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.221 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.222 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.579 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.066 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.181 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.241 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.242 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.590 31 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea6b94c1-3a5c-4bbb-91eb-090f8b681093] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:08.688 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:09.700 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:09.824 26 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:10.714 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [278de162-43c0-4950-8720-7fd2722136ef] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.244 30 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.475 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.668 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.486 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.681 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.498 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [79991de3-18a2-4fa5-87a2-4634c2203098] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.692 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [f443e6ba-0e77-4065-9b1d-a6515487e475] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.837 31 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.395 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db [req-354e97f9-5253-439b-b11d-f15fa6e82d0b - - - - -] Exception encountered during network rescheduling: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db Traceback (most recent call last): >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 291, in remove_networks_from_down_agents >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db context, cutoff) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/network.py", line 66, in get_down_bindings >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db agent_objs = agent_obj.Agent.get_objects(context) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 576, in get_objects >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 53, in get_objects >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db **(_pager.to_kwargs(context, obj_cls) if _pager else {})) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/neutron/db/_model_query.py", line 297, in get_collection >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db for c in query >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db close_with_result=True) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db **kw >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db conn = self.session.connection(**kw) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db execution_options=execution_options) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db engine, execution_options) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 398, in _connection_for_bind >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db conn = self._parent._connection_for_bind(bind, execution_options) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db conn = bind.contextual_connect() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db e, dialect, self) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return fn() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db rec.checkin() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db dbapi_connection = rec.get_connection() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db self.__connect() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db connection = pool._invoke_creator(self) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db return Connection(*args, **kwargs) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db self.connect() >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db raise exc >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 31 ERROR neutron.db.agentschedulers_db >2018-07-16 10:58:17.690 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:18.392 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:18.543 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:19.405 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:22.407 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [dc56964f-d448-45d7-8952-8d65e5c11da8] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.702 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [e0179583-ff88-43c7-b6fa-20385aaf7005] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:24.555 28 ERROR oslo.messaging._drivers.impl_rabbit [-] [d8e67495-18ab-48b5-8801-d5c02e2c276e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.266 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1852351a-fb6d-49c4-9d26-79d5f9b8ab1a] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.267 30 ERROR oslo.messaging._drivers.impl_rabbit [-] [1ab42e8d-a17d-4f7e-927a-4e33613da68c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.262 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [87a9e641-b5f2-477c-9c0d-e85e41ee02ac] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.352 27 ERROR oslo.messaging._drivers.impl_rabbit [-] [916ef3c0-98ac-491a-a78d-10af819cc3b4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.587 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b1b13ca-6c93-4e5b-b2fa-29fa3c56c266] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:25.686 29 ERROR oslo.messaging._drivers.impl_rabbit [-] [efea24a8-5ee4-459c-be1a-f10e3ca6681e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 20 seconds. Client port: None: timeout: timed out >2018-07-16 11:04:54.456 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:04.462 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:14.470 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:25.376 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:35.386 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:45.399 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:55.412 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:05.421 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:15.431 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:25.442 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.453 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -11 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:45.459 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -12 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:55.470 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -13 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:05.480 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -14 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:15.490 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -15 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:25.495 1 WARNING oslo_db.sqlalchemy.engines [req-e3c3bceb-8649-40e8-b9db-40498ccba9d4 - - - - -] SQL connection failed. -16 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:10:25.571 82 WARNING neutron.plugins.ml2.drivers.type_tunnel [req-ac124426-fbbe-4a37-b755-4c1bca71bc32 - - - - -] Endpoint with ip 10.10.126.109 already exists: DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '10.10.126.109' for key 'PRIMARY'") [SQL: u'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameters: {'host': u'controller-1.localdomain', 'ip_address': u'10.10.126.109', 'udp_port': 4789}] (Background on this error at: http://sqlalche.me/e/gkpj)
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1459171">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1601472
:
1459162
| 1459171 |
1459172