Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1522872 - [UPDATES] Failed to spawn vm after OC upate Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info'
[UPDATES] Failed to spawn vm after OC upate Queue.declare: (404) NOT_FOUND - ...
Status: ASSIGNED
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: ---
: 12.0 (Pike)
Assigned To: Dan Macpherson
RHOS Documentation Team
: Reopened
: 1550210 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-12-06 10:59 EST by Yurii Prokulevych
Modified: 2018-11-02 18:04 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
OpenStack Compute (nova) provides both versioned and unversioned notifications in RabbitMQ. However, due to the lack of consumers for versioned notifications, the versioned notifications queue grows quickly and causes RabbitMQ failures. This can hinder Compute operations such as instance creation and flavor creation. Red Hat is currently implementing fixes for RabbitMQ and director: https://bugzilla.redhat.com/show_bug.cgi?id=1478274 https://bugzilla.redhat.com/show_bug.cgi?id=1488499 The following article provides a workaround until Red Hat releases patches for this issue: https://access.redhat.com/solutions/3139721
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-08 10:13:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yurii Prokulevych 2017-12-06 10:59:26 EST
Description of problem:
-----------------------
After RHOS-12 minor update failed to spawn new vm. In conductor.log got next traceback:
2017-12-06 13:22:49.881 24 ERROR oslo.messaging._drivers.impl_rabbit [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Failed to publi
sh message to topic 'nova': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to 
perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] Could not send noti
fication to versioned_notifications. Payload={'_context_domain': None, '_context_roles': [u'admin'], '_context_global_request_id': None, '_context_quota_class': None, 'event_type': u'instance.update', '_context_
request_id': u'req-a8ce222c-010a-4a27-80e9-8da0972d6278', '_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'region': u'reg
ionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v2/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v2/358cf3d490424818a6365fc4379637a0'}], u'type
': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd0
0:2000::16]:8776/v1/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v1/358cf3d490424818a6365fc4379637a0'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints'
: [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'region': u'regionOne', u'internalURL': u'http://[fd00:fd00:fd00:2000::16]:8778/placement', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:f
e3e:1]:13778/placement'}], u'type': u'placement', u'name': u'placement'}, {u'endpoints': [{u'adminURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'region': u'regionOne', u'in
ternalURL': u'http://[fd00:fd00:fd00:2000::16]:8776/v3/358cf3d490424818a6365fc4379637a0', u'publicURL': u'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13776/v3/358cf3d490424818a6365fc4379637a0'}], u'type': u'volumev3
', u'name': u'cinderv3'}], 'timestamp': u'2017-12-06 13:21:46.885360', '_context_user': u'3f51df1556e748ce82d6e12188baefee', '_unique_id': '91f739fd62a94d65a95e488cb8676299', '_context_resource_uuid': None, '_co
ntext_instance_lock_checked': False, '_context_is_admin_project': True, '_context_user_id': u'3f51df1556e748ce82d6e12188baefee', 'payload': {'nova_object.version': '1.6', 'nova_object.name': 'InstanceUpdatePaylo
ad', 'nova_object.namespace': 'nova', 'nova_object.data': {'availability_zone': u'nova', 'terminated_at': None, 'ip_addresses': [], 'ramdisk_id': u'', 'updated_at': None, 'state_update': {'nova_object.version': 
'1.0', 'nova_object.name': 'InstanceStateUpdatePayload', 'nova_object.namespace': 'nova', 'nova_object.data': {'old_state': u'building', 'state': u'building', 'old_task_state': u'scheduling', 'new_task_state': u
'scheduling'}}, 'bandwidth': [], 'old_display_name': None, 'image_uuid': u'51672902-0005-42b8-922e-9649506799f4', 'flavor': {'nova_object.version': '1.3', 'nova_object.name': 'FlavorPayload', 'nova_object.namesp
ace': 'nova', 'nova_object.data': {'memory_mb': 128, 'root_gb': 0, 'name': u'm1.micro', 'ephemeral_gb': 0, 'disabled': False, 'vcpus': 1, 'extra_specs': {}, 'swap': 0, 'rxtx_factor': 1.0, 'is_public': True, 'fla
vorid': u'09363262-fd58-472d-bebb-0e4dc0405b77', 'vcpu_weight': 0, 'projects': None}}, 'deleted_at': None, 'reservation_id': u'r-6037twbk', 'user_id': u'3f51df1556e748ce82d6e12188baefee', 'uuid': '29eac6da-bcd1-
4b55-87a2-17f5868d6072', 'display_description': u'post-update-b4-reboot', 'state': u'building', 'power_state': u'pending', 'host_name': u'post-update-b4-reboot', 'progress': 0, 'launched_at': None, 'metadata': {
}, 'node': None, 'tags': [], 'kernel_id': u'', 'key_name': u'userkey', 'host': None, 'display_name': u'post-update-b4-reboot', 'task_state': u'scheduling', 'locked': False, 'tenant_id': u'358cf3d490424818a6365fc
4379637a0', 'created_at': '2017-12-06T13:21:44Z', 'block_devices': None, 'architecture': None, 'audit_period': {'nova_object.version': '1.0', 'nova_object.name': 'AuditPeriodPayload', 'nova_object.namespace': 'n
ova', 'nova_object.data': {'audit_period_ending': '2017-12-06T13:21:46Z', 'audit_period_beginning': '2017-12-01T00:00:00Z'}}, 'auto_disk_config': u'MANUAL', 'os_type': None}}, '_context_project_name': u'admin', 
'_context_read_deleted': u'no', '_context_user_identity': u'3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default', '_context_auth_token': u'gAAAAABaJ-7iwlHT0qzrIatuzRvxK9n2j6cRdoSR
Da-8sBaICio8giGn3UHsJZqN--ke_eGbCrKPajy3EO8fRjyMUea8XxjsbtsRZoptOZzTUAu0gLUH5JdMDqP_R8kGDDy7oh39_qWV5OM67oqfa_CJ1ngk1_G5UqhSc9Lf4ju-5gQR_8vGxtU', '_context_show_deleted': False, '_context_tenant': u'358cf3d49042
4818a6365fc4379637a0', 'priority': 'INFO', '_context_read_only': False, '_context_is_admin': True, '_context_project_id': u'358cf3d490424818a6365fc4379637a0', '_context_project_domain': u'default', '_context_timestamp': '2017-12-06T13:21:42.084054', '_context_user_domain': u'default', '_context_user_name': u'admin', 'publisher_id': u'nova-conductor:controller-0.localdomain', 'message_id': u'591d3ba8-82bd-4d33-b0a9-1c821974be71', '_context_remote_address': u'fd00:fd00:fd00:2000::15'}: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     retry=retry)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 130, in _send_notification
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     retry=retry)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 582, in send_notification
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 547, in _send
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     conn.notify_send(exchange, target.topic, msg, retry=retry)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1297, in notify_send
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     exchange, msg, routing_key=topic, retry=retry)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     self.ensure(method, retry=retry, error_callback=_error_callback)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     ret, channel = autoretry_method()
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     return fun(*args, **kwargs)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     return fun(*args, channel=channels[0], **kwargs), channels[0]
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     method()
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1225, in _publish_and_creates_default_queue
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     queue.declare()
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 605, in declare
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     self._create_queue(nowait=nowait, channel=channel)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 614, in _create_queue
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     self.queue_declare(nowait=nowait, passive=False, channel=channel)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 649, in queue_declare
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     nowait=nowait,
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1165, in queue_declare
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     spec.Queue.DeclareOk, returns_tuple=True,
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 93, in wait
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     self.connection.drain_events(timeout=timeout)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 465, in drain_events
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     return self.blocking_read(timeout)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 470, in blocking_read
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     return self.on_inbound_frame(frame)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     callback(channel, method_sig, buf, None)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 474, in on_inbound_method
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     method_sig, payload, content,
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     listener(*args)
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 293, in _on_close
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging     reply_code, reply_text, (class_id, method_id), ChannelError,
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.info' in vhost '/' due to timeout
2017-12-06 13:22:49.882 24 ERROR oslo_messaging.notify.messaging
2017-12-06 13:22:49.891 24 DEBUG nova.conductor.manager [req-a8ce222c-010a-4a27-80e9-8da0972d6278 3f51df1556e748ce82d6e12188baefee 358cf3d490424818a6365fc4379637a0 - default default] [instance: 29eac6da-bcd1-4b55-87a2-17f5868d6072] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='51672902-0005-42b8-922e-9649506799f4',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=None)] _create_block_device_mapping /usr/lib/python2.7/site-packages/nova/conductor/manager.py:936


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
Images from 12.0-20171201.1


Steps to Reproduce:
-------------------
1. Install UC/OC with 2017-11-29.2
2. Install repos from 2017-12-01.4
3. Update UC
4. Reboot UC
5. Run init-minor-update
6. Perform oc update starting with Controller nodes
7. Try to spawn VM after update is finished

Actual results:
---------------
Fail to launch vm.


Expected results:
-----------------
VM is launched


Additional info:
----------------
Virtual setup: 3controller + 2computes + 3ceph; uc/oc ssl
VM spawned before update with associated floating ip is reachable after update.
Comment 2 Yurii Prokulevych 2017-12-06 11:27:58 EST
Forgot to mention that overcloud deployed with IPv6
Comment 5 Dan Smith 2017-12-08 10:13:54 EST
This is a dupe of:

  https://bugzilla.redhat.com/show_bug.cgi?id=1455793

which is closed NOTABUG because the issue is actually in rabbit:

  https://bugzilla.redhat.com/show_bug.cgi?id=1488499

There is a workaround, which is to disable the versioned notifications in config so that this won't happen, as described here:

  https://access.redhat.com/solutions/3139721

So I'm closing this as NOTABUG as well.
Comment 8 Alexander Chuzhoy 2018-02-28 15:46:56 EST
*** Bug 1550210 has been marked as a duplicate of this bug. ***
Comment 9 Alexander Chuzhoy 2018-02-28 15:59:22 EST
Reproduced.
Re-opening and switching to documentation.

We should include the w/a in the kb[1] with the minor update documentation[2].


[1]

  https://access.redhat.com/solutions/3139721

[2]

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/upgrading_red_hat_openstack_platform/#keeping_openstack_platform_updated
Comment 10 Dan Macpherson 2018-08-09 02:24:29 EDT
Scoping old BZs.

The issue appears to be resolved for OSP12. Sasha, can you verify if this is so? Or do we still need a note?
Comment 11 Alexander Chuzhoy 2018-08-13 09:24:37 EDT
Yurii,
Is the note still needed?

Note You need to log in before you can comment on or make changes to this bug.