OpenStack Compute (nova) provides both versioned and unversioned notifications in RabbitMQ. However, due to the lack of consumers for versioned notifications, the versioned notifications queue grows quickly and causes RabbitMQ failures. This can hinder Compute operations such as instance creation and flavor creation. Red Hat is currently implementing fixes for RabbitMQ and director:
https://bugzilla.redhat.com/show_bug.cgi?id=1478274
https://bugzilla.redhat.com/show_bug.cgi?id=1488499
The following article provides a workaround until Red Hat releases patches for this issue:
https://access.redhat.com/solutions/3139721
Description of problem: ----------------------- After RHOS-12 minor update failed to spawn new vm. In conductor.log got next traceback: 2017-12-06 13:22:49.881 24 ERROR oslo.messaging._drivers.impl_rabbit [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Failed to publi sh message to topic 'nova': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Could not send noti fication to versioned_notifications. Payload={'_context_domain': None, '_context_roles': [u'admin'], '_context_global_request_id': None, '_context_quota_class': None, 'event_type': u'instance.update', '_context_ request_id': u'req-a8ce222c-010a-4a27-80e9-8da0972d6278', '_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'region': u'reg ionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v2/358cf3d490424818a6365fc4379637a0'}], u'type ': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd0 0:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v1/358cf3d490424818a6365fc4379637a0'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints' : [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:f e3e:1]:13778/placement'}], u'type': u'placement', u'name': u'placement'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'in ternalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v3/358cf3d490424818a6365fc4379637a0'}], u'type': u'volumev3 ', u'name': u'cinderv3'}], 'timestamp': u'2017-12-06 13:21:46.885360', '_context_user': u'3f51df1556e748ce82d6e12188baefee', '_unique_id': '91f739fd62a94d65a95e488cb8676299', '_context_resource_uuid': None, '_co ntext_instance_lock_checked': False, '_context_is_admin_project': True, '_context_user_id': u'3f51df1556e748ce82d6e12188baefee', 'payload': {'nova_object.version': '1.6', 'nova_object.name': 'InstanceUpdatePaylo ad', 'nova_object.namespace': 'nova', 'nova_object.data': {'availability_zone': u'nova', 'terminated_at': None, 'ip_addresses': [], 'ramdisk_id': u'', 'updated_at': None, 'state_update': {'nova_object.version': '1.0', 'nova_object.name': 'InstanceStateUpdatePayload', 'nova_object.namespace': 'nova', 'nova_object.data': {'old_state': u'building', 'state': u'building', 'old_task_state': u'scheduling', 'new_task_state': u 'scheduling'}}, 'bandwidth': [], 'old_display_name': None, 'image_uuid': u'51672902-0005-42b8-922e-9649506799f4', 'flavor': {'nova_object.version': '1.3', 'nova_object.name': 'FlavorPayload', 'nova_object.namesp ace': 'nova', 'nova_object.data': {'memory_mb': 128, 'root_gb': 0, 'name': u'm1.micro', 'ephemeral_gb': 0, 'disabled': False, 'vcpus': 1, 'extra_specs': {}, 'swap': 0, 'rxtx_factor': 1.0, 'is_public': True, 'fla vorid': u'09363262-fd58-472d-bebb-0e4dc0405b77', 'vcpu_weight': 0, 'projects': None}}, 'deleted_at': None, 'reservation_id': u'r-6037twbk', 'user_id': u'3f51df1556e748ce82d6e12188baefee', 'uuid': '29eac6da-bcd1- 4b55-87a2-17f5868d6072', 'display_description': u'post-update-b4-reboot', 'state': u'building', 'power_state': u'pending', 'host_name': u'post-update-b4-reboot', 'progress': 0, 'launched_at': None, 'metadata': { }, 'node': None, 'tags': [], 'kernel_id': u'', 'key_name': u'userkey', 'host': None, 'display_name': u'post-update-b4-reboot', 'task_state': u'scheduling', 'locked': False, 'tenant_id': u'358cf3d490424818a6365fc 4379637a0', 'created_at': '2017-12-06T13:21:44Z', 'block_devices': None, 'architecture': None, 'audit_period': {'nova_object.version': '1.0', 'nova_object.name': 'AuditPeriodPayload', 'nova_object.namespace': 'n ova', 'nova_object.data': {'audit_period_ending': '2017-12-06T13:21:46Z', 'audit_period_beginning': '2017-12-01T00:00:00Z'}}, 'auto_disk_config': u'MANUAL', 'os_type': None}}, '_context_project_name': u'admin', '_context_read_deleted': u'no', '_context_user_identity': u'3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default', '_context_auth_token': u'gAAAAABaJ-7iwlHT0qzrIatuzRvxK9n2j6cRdoSR Da-8sBaICio8giGn3UHsJZqN--ke_eGbCrKPajy3EO8fRjyMUea8XxjsbtsRZoptOZzTUAu0gLUH5JdMDqP_R8kGDDy7oh39_qWV5OM67oqfa_CJ1ngk1_G5UqhSc9Lf4ju-5gQR_8vGxtU', '_context_show_deleted': False, '_context_tenant': u'358cf3d49042 4818a6365fc4379637a0', 'priority': 'INFO', '_context_read_only': False, '_context_is_admin': True, '_context_project_id': u'358cf3d490424818a6365fc4379637a0', '_context_project_domain': u'default', '_context_timestamp': '2017-12-06T13:21:42.084054', '_context_user_domain': u'default', '_context_user_name': u'admin', 'publisher_id': u'nova-conductor:controller-0.localdomain', 'message_id': u'591d3ba8-82bd-4d33-b0a9-1c821974be71', '_context_remote_address': u'fd00:fd00:fd00:2000::15'}: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging Traceback (most recent call last): 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 130, in _send_notification 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 582, in send_notification 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging envelope=(version == 2.0), notify=True, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 547, in _send 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging conn.notify_send(exchange, target.topic, msg, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1297, in notify_send 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging exchange, msg, routing_key=topic, retry=retry) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.ensure(method, retry=retry, error_callback=_error_callback) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging ret, channel = autoretry_method() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return fun(*args, **kwargs) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return fun(*args, channel=channels[0], **kwargs), channels[0] 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging method() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1225, in _publish_and_creates_default_queue 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging queue.declare() 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 605, in declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self._create_queue(nowait=nowait, channel=channel) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 614, in _create_queue 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.queue_declare(nowait=nowait, passive=False, channel=channel) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 649, in queue_declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging nowait=nowait, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1165, in queue_declare 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging spec.Queue.DeclareOk, returns_tuple=True, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 93, in wait 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging self.connection.drain_events(timeout=timeout) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 465, in drain_events 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return self.blocking_read(timeout) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 470, in blocking_read 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging return self.on_inbound_frame(frame) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging callback(channel, method_sig, buf, None) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 474, in on_inbound_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging method_sig, payload, content, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging listener(*args) 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 293, in _on_close 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging reply_code, reply_text, (class_id, method_id), ChannelError, 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout 2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging 2017-12-06 13:22:49.891 24 DEBUG nova.conductor.manager [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] [instance: 29eac6da-bcd1-4b55-87a2-17f5868d6072] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='51672902-0005-42b8-922e-9649506799f4',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=None)] _create_block_device_mapping /usr/lib/python2.7/site-packages/nova/conductor/manager.py:936 Version-Release number of selected component (if applicable): ------------------------------------------------------------- Images from 12.0-20171201.1 Steps to Reproduce: ------------------- 1. Install UC/OC with 2017-11-29.2 2. Install repos from 2017-12-01.4 3. Update UC 4. Reboot UC 5. Run init-minor-update 6. Perform oc update starting with Controller nodes 7. Try to spawn VM after update is finished Actual results: --------------- Fail to launch vm. Expected results: ----------------- VM is launched Additional info: ---------------- Virtual setup: 3controller + 2computes + 3ceph; uc/oc ssl VM spawned before update with associated floating ip is reachable after update.