Bug 1277529
| Summary: | Socket closed in nova: oslo.messaging._drivers.impl_rabbit IOError: Socket closed | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ondrej <ochalups> |
| Component: | python-oslo-messaging | Assignee: | Flavio Percoco <fpercoco> |
| Status: | CLOSED DUPLICATE | QA Contact: | yeylon <yeylon> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.0 (Juno) | CC: | apevec, fpercoco, jeckersb, lhh, ochalups, plemenko, srevivo, yeylon |
| Target Milestone: | --- | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-25 16:47:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This looks familiar. I suspect that it's another issue with race condition between exchange creation and actual usage. Let me check. Just wondering what's your python-oslo-messaging version? Flavio, this looks similar to bug 1276166. What do you think? Hi Peter, here are python-oslo packages: python-oslo-config-1.4.0-1.el7ost.noarch python-oslo-db-1.0.2-2.el7ost.noarch python-oslo-i18n-1.0.0-1.el7ost.noarch python-oslo-messaging-1.4.1-5.el7ost.noarch python-oslo-rootwrap-1.3.0.0-2.el7ost.noarch python-oslo-serialization-0.3.0-1.el7ost.noarch python-oslo-utils-1.0.0-1.el7ost.noarch python-oslo-vmware-0.6.0-1.el7ost.noarch Thank you Ondrej (In reply to Ondrej from comment #5) > python-oslo-messaging-1.4.1-5.el7ost.noarch That's a slightly outdated version. Could you please upgrade to the latest 1.4.1-6? Unfortunately this won't solve the issue you're experiencing but will fix another one. Regarding the issue - as far as I see, that's a known issue in oslo.messaging. See this link for further details: * https://bugs.launchpad.net/nova/+bug/1460652 Upstream has a fix which yet to be backported: * https://review.openstack.org/#/c/193037/ I'm going to reassign this to oslo.messaging. Ok, closing this as a duplicate of bug 1276166. *** This bug has been marked as a duplicate of bug 1276166 *** |
Description of problem: instance creation failes with Timed out waiting for a reply to message ID node2 nova-conductor.log 2015-10-26 09:20:34.967 108508 ERROR oslo.messaging._drivers.impl_rabbit [req-4c456030-aa03-4b34-966a-82f928ec65a7 ] Failed to consume message from queue: Socket closed 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 661, in ensure 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 749, in _consume 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=poll_timeout) 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 302, in drain_events 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout, 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 365, in _wait_multiple 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout) 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 336, in read_timeout 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method() 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit raise m 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed 2015-10-26 09:20:34.967 108508 TRACE oslo.messaging._drivers.impl_rabbit 2015-10-26 09:20:34.968 108508 INFO oslo.messaging._drivers.impl_rabbit [req-4c456030-aa03-4b34-966a-82f928ec65a7 ] Delaying reconnect for 1.0 seconds... 2015-10-26 09:20:35.969 108508 INFO oslo.messaging._drivers.impl_rabbit [req-4c456030-aa03-4b34-966a-82f928ec65a7 ] Connecting to AMQP server on 10.24.82.111:5672 2015-10-26 09:20:35.983 108508 INFO oslo.messaging._drivers.impl_rabbit [req-4c456030-aa03-4b34-966a-82f928ec65a7 ] Connected to AMQP server on 10.24.82.111:5672 2015-10-26 14:37:36.151 108512 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_3803576fdbda49eb897114be623bc296': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_3803576fdbda49eb897114be623bc296' in vhost '/' 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 661, in ensure 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 769, in _publish 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__ 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_3803576fdbda49eb897114be623bc296' in vhost '/' 2015-10-26 14:37:36.151 108512 TRACE oslo.messaging._drivers.impl_rabbit 2015-10-26 14:37:36.152 108512 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-10-26 14:37:37.153 108512 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.24.82.110:5672 2015-10-26 14:37:37.165 108512 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.24.82.110:5672 node2 nova-scheduler.log 2015-10-26 14:37:36.143 108068 ERROR oslo.messaging._drivers.impl_rabbit [req-63972950-9c49-4b82-980e-aaf8034a9113 ] Failed to consume message from queue: Socket closed 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 661, in ensure 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 749, in _consume 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=poll_timeout) 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 302, in drain_events 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout, 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 365, in _wait_multiple 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout) 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 336, in read_timeout 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method() 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit raise m 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed 2015-10-26 14:37:36.143 108068 TRACE oslo.messaging._drivers.impl_rabbit 2015-10-26 14:37:36.145 108068 INFO oslo.messaging._drivers.impl_rabbit [req-63972950-9c49-4b82-980e-aaf8034a9113 ] Delaying reconnect for 1.0 seconds... 2015-10-26 14:37:37.146 108068 INFO oslo.messaging._drivers.impl_rabbit [req-63972950-9c49-4b82-980e-aaf8034a9113 ] Connecting to AMQP server on 10.24.82.110:5672 2015-10-26 14:37:37.167 108068 INFO oslo.messaging._drivers.impl_rabbit [req-63972950-9c49-4b82-980e-aaf8034a9113 ] Connected to AMQP server on 10.24.82.110:5672 2015-10-26 14:37:37.178 108068 ERROR nova.api.metadata.handler [req-63972950-9c49-4b82-980e-aaf8034a9113 None] Failed to get metadata for instance id: e8df6a33-6c3d-4897-b3aa-c6c92341dcc3 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler Traceback (most recent call last): 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 207, in _handle_instance_id_request 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler remote_address) 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 98, in get_metadata_by_instance_id 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler instance_id, address) 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/base.py", line 521, in get_metadata_by_instance_id 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler return InstanceMetadata(instance, address) 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/base.py", line 136, in __init__ 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler instance['host'], capi) 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/ec2/ec2utils.py", line 178, in get_availability_zone_by_host 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler context.get_admin_context(), host, conductor_api) 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/availability_zones.py", line 116, in get_host_availability_zone 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler if 'availability_zone' in metadata: 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler TypeError: argument of type 'NoneType' is not iterable 2015-10-26 14:37:37.178 108068 TRACE nova.api.metadata.handler 2015-10-26 14:37:37.179 108068 INFO nova.api.ec2 [-] 1.67598s 10.24.82.111 GET /openstack/latest/meta_data.json None:None 500 [Python-httplib2/0.7.7 (gzip)] text/plain text/plain 2015-10-26 14:37:37.180 108068 INFO nova.metadata.wsgi.server [-] 10.24.25.155,10.24.82.111 "GET /openstack/latest/meta_data.json HTTP/1.1" status: 500 len: 229 time: 1.0682528 node2 rabbit =ERROR REPORT==== 26-Oct-2015::14:37:36 === connection <0.5685.0>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_3803576fdbda49eb897114be623bc296' in vhost '/'", 'exchange.declare'} node1 nova-conductor.log 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 661, in ensure 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 769, in _publish 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__ 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 613, in exchange_declare 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2015-10-24 20:50:55.438 33650 TRACE oslo.messaging._drivers.impl_rabbit 2015-10-24 20:50:55.445 33650 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-10-24 20:50:56.446 33650 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.24.82.110:5672 2015-10-24 20:50:56.466 33650 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.24.82.110:5672 Version-Release number of selected component (if applicable): rabbitmq-server-3.3.5-4.el7ost.noarch openstack-nova-api-2014.2.3-25.el7ost.noarch openstack-nova-cert-2014.2.3-25.el7ost.noarch openstack-nova-common-2014.2.3-25.el7ost.noarch openstack-nova-compute-2014.2.3-25.el7ost.noarch openstack-nova-conductor-2014.2.3-25.el7ost.noarch openstack-nova-console-2014.2.3-25.el7ost.noarch openstack-nova-novncproxy-2014.2.3-25.el7ost.noarch openstack-nova-scheduler-2014.2.3-25.el7ost.noarch How reproducible: not everytime, once in a while Steps to Reproduce: 1.create an instance 2. 3. Actual results: request fails due to timeout waiting for reply Expected results: instance created Additional info: